[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:52:50 dist-test-slave-m9g1 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:52:50 dist-test-slave-m9g1 krb5kdc[395](info): set up 2 sockets
Sep 29 20:52:50 dist-test-slave-m9g1 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 20:52:51.839100 388 mini_kdc.cc:121] Time spent starting KDC: real 2.318s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:52:51 dist-test-slave-m9g1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:52:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:52:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:52:51.997992 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37769
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:52:52.305178 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:52:52.305472 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:52:52.305619 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:52:52.312803 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:52:52.312922 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:52:52.312999 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:52:52.313079 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:52:52.313141 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:52:52.323596 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:52:52.324891 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:52:52.326239 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:52:52.337970 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:52.338001 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:52.338039 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:52.338291 408 server_base.cc:1034] running on GCE node
I20260929 20:52:52.339272 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:52:52.340674 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:52:52.341872 408 hybrid_clock.cc:648] HybridClock initialized: now 1790715172341796 us; error 78 us; skew 500 ppm
I20260929 20:52:52.342242 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:52:52 dist-test-slave-m9g1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:52:52.345111 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:52:52.346542 408 webserver.cc:460] Webserver started at http://127.0.97.62:42473/ using document root <none> and password file <none>
I20260929 20:52:52.347177 408 fs_manager.cc:359] Metadata directory not provided
I20260929 20:52:52.347267 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:52:52.347498 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:52:52.350333 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "caf26e6a7f4f48fc8cd85fd0ddc5ef8a"
format_stamp: "Formatted at 2026-09-29 20:52:52 on dist-test-slave-m9g1"
server_key: "e291cf5dbe157a0e789f33fb09211318"
server_key_iv: "31055d16e0fecb20019a6ca4246c05fa"
server_key_version: "encryptionkey@0"
I20260929 20:52:52.351033 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "caf26e6a7f4f48fc8cd85fd0ddc5ef8a"
format_stamp: "Formatted at 2026-09-29 20:52:52 on dist-test-slave-m9g1"
server_key: "e291cf5dbe157a0e789f33fb09211318"
server_key_iv: "31055d16e0fecb20019a6ca4246c05fa"
server_key_version: "encryptionkey@0"
I20260929 20:52:52.358158 408 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260929 20:52:52.362092 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:52.363454 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:52:52.363607 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "caf26e6a7f4f48fc8cd85fd0ddc5ef8a"
format_stamp: "Formatted at 2026-09-29 20:52:52 on dist-test-slave-m9g1"
server_key: "e291cf5dbe157a0e789f33fb09211318"
server_key_iv: "31055d16e0fecb20019a6ca4246c05fa"
server_key_version: "encryptionkey@0"
I20260929 20:52:52.363794 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:52:52.377795 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:52:52.381690 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:52:52.381932 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:52:52.398701 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37769
I20260929 20:52:52.398728 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37769 every 8 connection(s)
I20260929 20:52:52.399991 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:52:52.403434 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:52:52.410638 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 408
I20260929 20:52:52.410890 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:52:52.412577 388 external_mini_cluster.cc:1428] Setting key c8bbe577943f502452b519d1230b3932
I20260929 20:52:52.413064 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a: Bootstrap starting.
I20260929 20:52:52.416286 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a: Neither blocks nor log segments found. Creating new log.
I20260929 20:52:52.417575 475 log.cc:824] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:52:52.421012 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a: No bootstrap required, opened a new log
I20260929 20:52:52.431739 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf26e6a7f4f48fc8cd85fd0ddc5ef8a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37769 } }
I20260929 20:52:52.432238 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:52:52.432358 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf26e6a7f4f48fc8cd85fd0ddc5ef8a, State: Initialized, Role: FOLLOWER
I20260929 20:52:52.433048 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf26e6a7f4f48fc8cd85fd0ddc5ef8a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37769 } }
I20260929 20:52:52.433313 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:52:52.433440 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:52:52.433611 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:52:52.436538 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf26e6a7f4f48fc8cd85fd0ddc5ef8a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37769 } }
I20260929 20:52:52.437031 475 leader_election.cc:302] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caf26e6a7f4f48fc8cd85fd0ddc5ef8a; no voters:
I20260929 20:52:52.437633 475 leader_election.cc:288] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:52:52.437724 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:52:52.439769 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:52:52.440026 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a [term 1 LEADER]: Becoming Leader. State: Replica: caf26e6a7f4f48fc8cd85fd0ddc5ef8a, State: Running, Role: LEADER
I20260929 20:52:52.440603 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf26e6a7f4f48fc8cd85fd0ddc5ef8a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37769 } }
I20260929 20:52:52.443670 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader caf26e6a7f4f48fc8cd85fd0ddc5ef8a. Latest consensus state: current_term: 1 leader_uuid: "caf26e6a7f4f48fc8cd85fd0ddc5ef8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf26e6a7f4f48fc8cd85fd0ddc5ef8a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37769 } } }
I20260929 20:52:52.444033 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a [sys.catalog]: This master's current role is: LEADER
Sep 29 20:52:52 dist-test-slave-m9g1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:52:52.447748 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caf26e6a7f4f48fc8cd85fd0ddc5ef8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf26e6a7f4f48fc8cd85fd0ddc5ef8a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37769 } } }
I20260929 20:52:52.448032 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a [sys.catalog]: This master's current role is: LEADER
I20260929 20:52:52.449590 489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:52:52.455485 489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:52:52.461434 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:52.415869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39150 (local address 127.0.97.62:37769)
0929 20:52:52.416754 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:52.416769 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:52.418004 (+ 1235us) server_negotiation.cc:407] Connection header received
0929 20:52:52.431079 (+ 13075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:52.431106 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:52.431550 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:52.431910 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:52.435969 (+ 4059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:52.437344 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:52.439786 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:52.440898 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:52.441077 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:52.449488 (+ 8411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:52.449500 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:52.449521 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:52.449571 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:52.454145 (+ 4574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:52.454952 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:52.454960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:52.454969 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:52.455187 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:52.455698 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:52.455701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:52.455703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:52.456216 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:52.456516 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:52.459490 (+ 2974us) server_negotiation.cc:299] Negotiation successful
0929 20:52:52.459953 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":377,"threads_started":1}
I20260929 20:52:52.469883 489 catalog_manager.cc:1347] Generated new cluster ID: 1452e9e54b58487bb9a781a8e50df251
I20260929 20:52:52.469985 489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:52:52.498553 489 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:52:52.499845 489 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:52:52.512006 489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P caf26e6a7f4f48fc8cd85fd0ddc5ef8a: Generated new TSK 0
I20260929 20:52:52.512892 489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:52:52.565623 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37769
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:52:52.884794 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:52:52.885088 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:52:52.885222 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:52:52.892549 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:52:52.892680 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:52:52.892843 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:52:52.902974 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37769
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:52:52.904203 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:52:52.905633 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:52:52.917804 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:52.917894 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:52.917881 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:52.918550 501 server_base.cc:1034] running on GCE node
I20260929 20:52:52.919113 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:52:52.919983 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:52:52.921172 501 hybrid_clock.cc:648] HybridClock initialized: now 1790715172921129 us; error 34 us; skew 500 ppm
I20260929 20:52:52.921481 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:52:52 dist-test-slave-m9g1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:52:52.924710 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:52:52.926079 501 webserver.cc:460] Webserver started at http://127.0.97.1:46649/ using document root <none> and password file <none>
I20260929 20:52:52.926750 501 fs_manager.cc:359] Metadata directory not provided
I20260929 20:52:52.926862 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:52:52.927179 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:52:52.929999 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "182f4435e6cb41ed80353e697f5057cb"
format_stamp: "Formatted at 2026-09-29 20:52:52 on dist-test-slave-m9g1"
server_key: "4585bbcc86c162100c52ab8e32680e97"
server_key_iv: "541a00eaaa08420f531618b3e2b85577"
server_key_version: "encryptionkey@0"
I20260929 20:52:52.930811 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "182f4435e6cb41ed80353e697f5057cb"
format_stamp: "Formatted at 2026-09-29 20:52:52 on dist-test-slave-m9g1"
server_key: "4585bbcc86c162100c52ab8e32680e97"
server_key_iv: "541a00eaaa08420f531618b3e2b85577"
server_key_version: "encryptionkey@0"
I20260929 20:52:52.935643 501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 20:52:52.938856 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:52.940147 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 20:52:52.940311 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "182f4435e6cb41ed80353e697f5057cb"
format_stamp: "Formatted at 2026-09-29 20:52:52 on dist-test-slave-m9g1"
server_key: "4585bbcc86c162100c52ab8e32680e97"
server_key_iv: "541a00eaaa08420f531618b3e2b85577"
server_key_version: "encryptionkey@0"
I20260929 20:52:52.940523 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:52:52.964742 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:52:52.968740 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:52:52.969002 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:52:52.969897 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:52:52.971177 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:52:52.971251 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:52.971366 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:52:52.971421 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:53.000746 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41799
I20260929 20:52:53.000754 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41799 every 8 connection(s)
I20260929 20:52:53.001938 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:52:53.011228 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 501
I20260929 20:52:53.011427 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:52:53.012042 388 external_mini_cluster.cc:1428] Setting key 6faf91e6aceb483a267881a4184224bd
Sep 29 20:52:53 dist-test-slave-m9g1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:52:53.021639 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:53.004672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41015 (local address 127.0.97.62:37769)
0929 20:52:53.005075 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:53.005080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:53.006388 (+ 1308us) server_negotiation.cc:407] Connection header received
0929 20:52:53.008108 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:53.008113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:53.008190 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:53.008280 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:53.009680 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:53.010434 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:53.011987 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:53.012456 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:53.012561 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:53.015957 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:53.015962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:53.015965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:53.015994 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:53.018382 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:53.019028 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:53.019033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:53.019035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:53.019150 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:53.019526 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:53.019531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:53.019535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:53.019786 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:53.019918 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:53.020554 (+ 636us) server_negotiation.cc:299] Negotiation successful
0929 20:52:53.020761 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":197,"threads_started":1}
I20260929 20:52:53.022367 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:53.005199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37769 (local address 127.0.97.1:41015)
0929 20:52:53.005835 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:53.005871 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:53.007866 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:53.008513 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:53.008521 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:53.009000 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:53.009531 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:53.009547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:53.010568 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:53.010572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:53.011863 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:53.011874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:53.012655 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:53.012662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:53.012863 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:53.013466 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:53.013487 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:53.015739 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:53.018504 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:53.018513 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:53.018530 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:53.018923 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:53.019243 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:53.019248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:53.019252 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:53.019430 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:53.019904 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:53.019911 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:53.020225 (+ 314us) client_negotiation.cc:769] Sending connection context
0929 20:52:53.020466 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 20:52:53.020979 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":242,"threads_started":1}
I20260929 20:52:53.023810 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37769
I20260929 20:52:53.024088 629 heartbeater.cc:464] Registering TS with master...
I20260929 20:52:53.024724 629 heartbeater.cc:511] Master 127.0.97.62:37769 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:52:53.026710 440 ts_manager.cc:194] Registered new tserver with Master: 182f4435e6cb41ed80353e697f5057cb (127.0.97.1:41799)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 20:52:53.028211 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:52:53.045266 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37769
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:52:53.358355 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:52:53.358614 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:52:53.358724 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:52:53.366117 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:52:53.366261 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:52:53.366392 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:52:53.375969 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37769
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:52:53.377173 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:52:53.378451 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:52:53.390717 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:53.390731 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:53.390882 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:53.391456 635 server_base.cc:1034] running on GCE node
I20260929 20:52:53.391996 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:52:53.392820 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:52:53.394006 635 hybrid_clock.cc:648] HybridClock initialized: now 1790715173393968 us; error 29 us; skew 500 ppm
I20260929 20:52:53.394317 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:52:53 dist-test-slave-m9g1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:52:53.400416 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:52:53.401779 635 webserver.cc:460] Webserver started at http://127.0.97.2:46717/ using document root <none> and password file <none>
I20260929 20:52:53.402436 635 fs_manager.cc:359] Metadata directory not provided
I20260929 20:52:53.402531 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:52:53.402801 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:52:53.405393 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "e50b228c5c18434f9bcbef4a94cbf9ad"
format_stamp: "Formatted at 2026-09-29 20:52:53 on dist-test-slave-m9g1"
server_key: "32613f514dbbf5d8b0214958a97676e4"
server_key_iv: "6bfe474b099398106169b689ea5cb44c"
server_key_version: "encryptionkey@0"
I20260929 20:52:53.406080 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e50b228c5c18434f9bcbef4a94cbf9ad"
format_stamp: "Formatted at 2026-09-29 20:52:53 on dist-test-slave-m9g1"
server_key: "32613f514dbbf5d8b0214958a97676e4"
server_key_iv: "6bfe474b099398106169b689ea5cb44c"
server_key_version: "encryptionkey@0"
I20260929 20:52:53.411646 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 20:52:53.415100 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:53.416601 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:52:53.416757 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "e50b228c5c18434f9bcbef4a94cbf9ad"
format_stamp: "Formatted at 2026-09-29 20:52:53 on dist-test-slave-m9g1"
server_key: "32613f514dbbf5d8b0214958a97676e4"
server_key_iv: "6bfe474b099398106169b689ea5cb44c"
server_key_version: "encryptionkey@0"
I20260929 20:52:53.416946 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:52:53.429590 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:52:53.433084 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:52:53.433308 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:52:53.434167 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:52:53.435896 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:52:53.435964 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:53.436050 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:52:53.436089 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:53.466143 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33103
I20260929 20:52:53.466176 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33103 every 8 connection(s)
I20260929 20:52:53.467482 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:52:53.468294 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 635
I20260929 20:52:53.468467 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:52:53.468986 388 external_mini_cluster.cc:1428] Setting key 184b157b6791dff29a0b6372835c5cce
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Sep 29 20:52:53 dist-test-slave-m9g1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:52:53.493813 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:53.470154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60825 (local address 127.0.97.62:37769)
0929 20:52:53.470366 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:53.470372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:53.476369 (+ 5997us) server_negotiation.cc:407] Connection header received
0929 20:52:53.478795 (+ 2426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:53.478800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:53.478878 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:53.478959 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:53.480864 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:53.481569 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:53.483806 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:53.484183 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:53.484268 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:53.488433 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:53.488438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:53.488441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:53.488473 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:53.491053 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:53.491730 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:53.491736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:53.491740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:53.491884 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:53.492238 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:53.492242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:53.492246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:53.492551 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:53.492695 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:53.493300 (+ 605us) server_negotiation.cc:299] Negotiation successful
0929 20:52:53.493512 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 20:52:53.494680 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:53.470702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37769 (local address 127.0.97.2:60825)
0929 20:52:53.476207 (+ 5505us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:53.476256 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:53.478523 (+ 2267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:53.479220 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:53.479231 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:53.479920 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:53.480704 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:53.480729 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:53.481712 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:53.481717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:53.483687 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:53.483700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:53.484386 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:53.484393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:53.484675 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:53.485314 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:53.485336 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:53.488222 (+ 2886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:53.491186 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:53.491193 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:53.491206 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:53.491596 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:53.491976 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:53.491979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:53.491982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:53.492146 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:53.492654 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:53.492660 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:53.492959 (+ 299us) client_negotiation.cc:769] Sending connection context
0929 20:52:53.493170 (+ 211us) client_negotiation.cc:240] Negotiation successful
0929 20:52:53.493537 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5199,"thread_start_us":5593,"threads_started":1}
I20260929 20:52:53.496155 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37769
I20260929 20:52:53.496459 763 heartbeater.cc:464] Registering TS with master...
I20260929 20:52:53.497145 763 heartbeater.cc:511] Master 127.0.97.62:37769 requested a full tablet report, sending...
I20260929 20:52:53.498536 440 ts_manager.cc:194] Registered new tserver with Master: e50b228c5c18434f9bcbef4a94cbf9ad (127.0.97.2:33103)
I20260929 20:52:53.499326 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:52:53.504910 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37769
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:52:53.812260 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:52:53.812551 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:52:53.812681 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:52:53.819921 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:52:53.820036 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:52:53.820188 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:52:53.830086 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37769
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:52:53.831284 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:52:53.832504 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:52:53.845438 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:53.845903 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:53.845935 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:53.846165 768 server_base.cc:1034] running on GCE node
I20260929 20:52:53.846822 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:52:53.847659 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:52:53.848842 768 hybrid_clock.cc:648] HybridClock initialized: now 1790715173848794 us; error 41 us; skew 500 ppm
I20260929 20:52:53.849131 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:52:53 dist-test-slave-m9g1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:52:53.851992 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:52:53.853294 768 webserver.cc:460] Webserver started at http://127.0.97.3:43821/ using document root <none> and password file <none>
I20260929 20:52:53.853945 768 fs_manager.cc:359] Metadata directory not provided
I20260929 20:52:53.854055 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:52:53.854365 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:52:53.857498 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "c3e45f12e7f0465ea045dabee8fc2732"
format_stamp: "Formatted at 2026-09-29 20:52:53 on dist-test-slave-m9g1"
server_key: "146cfbd99cc48762bd076284d21e86cb"
server_key_iv: "316792a85c919bf47fd2584389dea71e"
server_key_version: "encryptionkey@0"
I20260929 20:52:53.858301 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c3e45f12e7f0465ea045dabee8fc2732"
format_stamp: "Formatted at 2026-09-29 20:52:53 on dist-test-slave-m9g1"
server_key: "146cfbd99cc48762bd076284d21e86cb"
server_key_iv: "316792a85c919bf47fd2584389dea71e"
server_key_version: "encryptionkey@0"
I20260929 20:52:53.863086 768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 20:52:53.866232 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:53.867460 768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 20:52:53.867597 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "c3e45f12e7f0465ea045dabee8fc2732"
format_stamp: "Formatted at 2026-09-29 20:52:53 on dist-test-slave-m9g1"
server_key: "146cfbd99cc48762bd076284d21e86cb"
server_key_iv: "316792a85c919bf47fd2584389dea71e"
server_key_version: "encryptionkey@0"
I20260929 20:52:53.867758 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:52:53.884260 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:52:53.888162 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:52:53.888424 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:52:53.889715 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:52:53.891024 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:52:53.891096 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:53.891211 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:52:53.891261 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:53.923061 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32983
I20260929 20:52:53.923102 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32983 every 8 connection(s)
I20260929 20:52:53.924253 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:52:53.928090 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 768
I20260929 20:52:53.928277 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:52:53.928802 388 external_mini_cluster.cc:1428] Setting key 3e46d1f3b6eead48972d48aef834ace1
Sep 29 20:52:53 dist-test-slave-m9g1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:52:53.943568 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:53.926581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55729 (local address 127.0.97.62:37769)
0929 20:52:53.926756 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:53.926761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:53.927928 (+ 1167us) server_negotiation.cc:407] Connection header received
0929 20:52:53.930716 (+ 2788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:53.930720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:53.930796 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:53.930874 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:53.932619 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:53.933333 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:53.934957 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:53.935345 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:53.935435 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:53.938368 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:53.938373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:53.938376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:53.938407 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:53.940786 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:53.941428 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:53.941432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:53.941435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:53.941544 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:53.941931 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:53.941934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:53.941937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:53.942232 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:53.942356 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:53.943011 (+ 655us) server_negotiation.cc:299] Negotiation successful
0929 20:52:53.943211 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:52:53.944525 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:53.927039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37769 (local address 127.0.97.3:55729)
0929 20:52:53.927743 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:53.927790 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:53.930444 (+ 2654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:53.931133 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:53.931143 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:53.931817 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:53.932461 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:53.932484 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:53.933460 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:53.933464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:53.934847 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:53.934858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:53.935506 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:53.935510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:53.935692 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:53.936228 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:53.936251 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:53.938164 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:53.940911 (+ 2747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:53.940920 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:53.940935 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:53.941314 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:53.941637 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:53.941642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:53.941646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:53.941819 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:53.942341 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:53.942348 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:53.942647 (+ 299us) client_negotiation.cc:769] Sending connection context
0929 20:52:53.942888 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 20:52:53.943369 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":243,"threads_started":1}
I20260929 20:52:53.945716 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37769
I20260929 20:52:53.945986 896 heartbeater.cc:464] Registering TS with master...
I20260929 20:52:53.946659 896 heartbeater.cc:511] Master 127.0.97.62:37769 requested a full tablet report, sending...
I20260929 20:52:53.947889 440 ts_manager.cc:194] Registered new tserver with Master: c3e45f12e7f0465ea045dabee8fc2732 (127.0.97.3:32983)
I20260929 20:52:53.948694 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55729
I20260929 20:52:53.956398 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:52:53 dist-test-slave-m9g1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:52:53 dist-test-slave-m9g1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:52:53.995332 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:53.981833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45832 (local address 127.0.97.1:41799)
0929 20:52:53.982569 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:53.982576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:53.982603 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:52:53.982680 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:53.982686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:53.982894 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:53.983011 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:53.984238 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:53.985221 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:53.986428 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:53.986935 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:53.987025 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:53.990537 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:53.990548 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:53.990562 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:53.990594 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:53.993070 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:53.993637 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:53.993643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:53.993645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:53.993749 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:53.994034 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:53.994037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:53.994039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:53.994372 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:53.994596 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:53.994791 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 20:52:53.994982 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":482,"threads_started":1}
W20260929 20:52:53.996553 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45832
Sep 29 20:52:54 dist-test-slave-m9g1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:52:54 dist-test-slave-m9g1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:52:54.031036 629 heartbeater.cc:503] Master 127.0.97.62:37769 was elected leader, sending a full tablet report...
I20260929 20:52:54.033902 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:54.021821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45844 (local address 127.0.97.1:41799)
0929 20:52:54.022043 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:54.022049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:54.022278 (+ 229us) server_negotiation.cc:407] Connection header received
0929 20:52:54.022428 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:54.022433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:54.022538 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:54.022655 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:54.023814 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:54.024469 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:54.025540 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:54.025917 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:54.025999 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:54.029326 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:54.029331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:54.029334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:54.029364 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:54.031774 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:54.032437 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:54.032442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:54.032445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:54.032571 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:54.032890 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:54.032893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:54.032895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:54.033112 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:54.033262 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:54.033469 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 20:52:54.033614 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 20:52:54.035161 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 501
I20260929 20:52:54.046939 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:54.036000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.0.97.1:41799)
0929 20:52:54.036126 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:54.036130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:54.036851 (+ 721us) server_negotiation.cc:407] Connection header received
0929 20:52:54.036970 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:54.036973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:54.037021 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:54.037084 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:54.038022 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:54.038689 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:54.040074 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:54.040482 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:54.040573 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:54.042553 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:54.042557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:54.042560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:54.042586 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:54.044921 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:54.045468 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:54.045472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:54.045474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:54.045585 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:54.045842 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:54.045845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:54.045847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:54.046066 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:54.046251 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:54.046455 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 20:52:54.046611 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 20:52:54.047520 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:52:54.126560 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 501
I20260929 20:52:54.139621 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 635
Sep 29 20:52:54 dist-test-slave-m9g1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:52:54.153937 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:54.140519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43310 (local address 127.0.97.2:33103)
0929 20:52:54.140969 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:54.140976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:54.141002 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:52:54.141058 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:54.141064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:54.141308 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:54.141450 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:54.142564 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:54.143511 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:54.144673 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:54.145187 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:54.145295 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:54.148923 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:54.148933 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:54.148947 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:54.148985 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:54.151456 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:54.152029 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:54.152036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:54.152039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:54.152164 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:54.152521 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:54.152525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:54.152527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:54.152839 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:54.153118 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:54.153371 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 20:52:54.153615 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":207,"threads_started":1}
I20260929 20:52:54.155332 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:52:54.234293 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 635
I20260929 20:52:54.252261 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 768
Sep 29 20:52:54 dist-test-slave-m9g1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:52:54.267136 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:54.253122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.0.97.3:32983)
0929 20:52:54.253713 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:54.253720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:54.253748 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:52:54.253816 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:54.253822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:54.254028 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:54.254175 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:54.255379 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:54.256411 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:54.257613 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:54.258156 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:54.258299 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:54.261997 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:54.262008 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:54.262022 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:54.262057 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:54.264462 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:54.265032 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:54.265039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:54.265041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:54.265155 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:54.265488 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:54.265493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:54.265496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:54.265824 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:54.266094 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:54.266554 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 20:52:54.266811 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":290,"threads_started":1}
I20260929 20:52:54.268445 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:52:54.348726 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 768
I20260929 20:52:54.361868 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 408
I20260929 20:52:54.363003 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:52:54.461119 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 408
2026-09-29T20:52:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4986 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:52:54 dist-test-slave-m9g1 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:52:54 dist-test-slave-m9g1 krb5kdc[917](info): set up 2 sockets
Sep 29 20:52:54 dist-test-slave-m9g1 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260929 20:52:55.433775 388 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.001s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:52:55 dist-test-slave-m9g1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:52:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:52:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:52:55.589401 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35805
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:52:55.889820 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:52:55.890120 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:52:55.890271 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:52:55.897506 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:52:55.897634 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:52:55.897710 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:52:55.897782 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:52:55.897847 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:52:55.907756 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:52:55.909132 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:52:55.910434 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:52:55.921319 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:55.921379 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:55.921348 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:55.922324 930 server_base.cc:1034] running on GCE node
I20260929 20:52:55.923053 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:52:55.924369 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:52:55.925558 930 hybrid_clock.cc:648] HybridClock initialized: now 1790715175925508 us; error 37 us; skew 500 ppm
I20260929 20:52:55.925890 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:52:55 dist-test-slave-m9g1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:52:55.928709 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:52:55.930114 930 webserver.cc:460] Webserver started at http://127.0.97.62:41597/ using document root <none> and password file <none>
I20260929 20:52:55.930758 930 fs_manager.cc:359] Metadata directory not provided
I20260929 20:52:55.930867 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:52:55.931176 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:52:55.934166 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "6d4abe2a912d4dfaa1eb2e7824776e0a"
format_stamp: "Formatted at 2026-09-29 20:52:55 on dist-test-slave-m9g1"
server_key: "e3c136836995729337c24f2fd4321863"
server_key_iv: "ecc4cb908968b0d5ea962f5bc5286762"
server_key_version: "encryptionkey@0"
I20260929 20:52:55.934929 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "6d4abe2a912d4dfaa1eb2e7824776e0a"
format_stamp: "Formatted at 2026-09-29 20:52:55 on dist-test-slave-m9g1"
server_key: "e3c136836995729337c24f2fd4321863"
server_key_iv: "ecc4cb908968b0d5ea962f5bc5286762"
server_key_version: "encryptionkey@0"
I20260929 20:52:55.939651 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:52:55.942849 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:55.944048 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:52:55.944197 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "6d4abe2a912d4dfaa1eb2e7824776e0a"
format_stamp: "Formatted at 2026-09-29 20:52:55 on dist-test-slave-m9g1"
server_key: "e3c136836995729337c24f2fd4321863"
server_key_iv: "ecc4cb908968b0d5ea962f5bc5286762"
server_key_version: "encryptionkey@0"
I20260929 20:52:55.944356 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:52:55.961269 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:52:55.964987 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:52:55.965229 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:52:55.981259 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35805
I20260929 20:52:55.981295 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35805 every 8 connection(s)
I20260929 20:52:55.982590 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:52:55.986019 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:52:55.992825 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 930
I20260929 20:52:55.993045 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:52:55.993813 388 external_mini_cluster.cc:1428] Setting key c9eb1ca943bf58b91de86505fe183249
I20260929 20:52:55.997063 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a: Bootstrap starting.
I20260929 20:52:56.000488 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a: Neither blocks nor log segments found. Creating new log.
I20260929 20:52:56.001538 997 log.cc:824] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:52:56.004422 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a: No bootstrap required, opened a new log
Sep 29 20:52:56 dist-test-slave-m9g1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:52:56.014281 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4abe2a912d4dfaa1eb2e7824776e0a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35805 } }
I20260929 20:52:56.014739 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:52:56.014876 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4abe2a912d4dfaa1eb2e7824776e0a, State: Initialized, Role: FOLLOWER
I20260929 20:52:56.015587 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4abe2a912d4dfaa1eb2e7824776e0a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35805 } }
I20260929 20:52:56.015872 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:52:56.016001 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:52:56.016175 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:52:56.017223 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:55.995795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57176 (local address 127.0.97.62:35805)
0929 20:52:55.996529 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:55.996541 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:55.996581 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:52:55.998289 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:55.998310 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:55.998715 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:55.999024 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:56.000678 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:56.002008 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:56.003579 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:56.004203 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:56.004322 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:56.008764 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:56.008776 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:56.008795 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:56.008859 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:56.012771 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:56.013379 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:56.013387 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:56.013397 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:56.013593 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:56.013883 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:56.013887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:56.013890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:56.014628 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:56.014914 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:56.015392 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 20:52:56.015835 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":318,"threads_started":1}
I20260929 20:52:56.018843 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4abe2a912d4dfaa1eb2e7824776e0a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35805 } }
I20260929 20:52:56.019306 997 leader_election.cc:302] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d4abe2a912d4dfaa1eb2e7824776e0a; no voters:
I20260929 20:52:56.019882 997 leader_election.cc:288] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:52:56.019979 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:52:56.020538 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a [term 1 LEADER]: Becoming Leader. State: Replica: 6d4abe2a912d4dfaa1eb2e7824776e0a, State: Running, Role: LEADER
I20260929 20:52:56.021086 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4abe2a912d4dfaa1eb2e7824776e0a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35805 } }
I20260929 20:52:56.022053 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:52:56.023810 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d4abe2a912d4dfaa1eb2e7824776e0a. Latest consensus state: current_term: 1 leader_uuid: "6d4abe2a912d4dfaa1eb2e7824776e0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4abe2a912d4dfaa1eb2e7824776e0a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35805 } } }
I20260929 20:52:56.023898 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d4abe2a912d4dfaa1eb2e7824776e0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4abe2a912d4dfaa1eb2e7824776e0a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35805 } } }
I20260929 20:52:56.024176 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a [sys.catalog]: This master's current role is: LEADER
I20260929 20:52:56.024207 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a [sys.catalog]: This master's current role is: LEADER
I20260929 20:52:56.026048 1007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:52:56.031554 1007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:52:56.041183 1007 catalog_manager.cc:1347] Generated new cluster ID: 323ba6a438034a9c8bc4ffbc5e2b0cc1
I20260929 20:52:56.041321 1007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:52:56.052598 1007 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:52:56.054409 1007 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:52:56.073473 1007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d4abe2a912d4dfaa1eb2e7824776e0a: Generated new TSK 0
I20260929 20:52:56.074585 1007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:52:56.119634 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35805
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:52:56.423887 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:52:56.424157 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:52:56.424252 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:52:56.431258 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:52:56.431380 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:52:56.431540 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:52:56.441092 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35805
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:52:56.442452 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:52:56.443812 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:52:56.455859 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:56.455904 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:56.455941 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:56.456730 1023 server_base.cc:1034] running on GCE node
I20260929 20:52:56.457326 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:52:56.458225 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:52:56.459481 1023 hybrid_clock.cc:648] HybridClock initialized: now 1790715176459439 us; error 34 us; skew 500 ppm
I20260929 20:52:56.459774 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:52:56 dist-test-slave-m9g1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:52:56.462690 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:52:56.464077 1023 webserver.cc:460] Webserver started at http://127.0.97.1:33685/ using document root <none> and password file <none>
I20260929 20:52:56.464761 1023 fs_manager.cc:359] Metadata directory not provided
I20260929 20:52:56.464875 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:52:56.465169 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:52:56.468108 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "d7cdcda32e8a421db84a88f5b0a49554"
format_stamp: "Formatted at 2026-09-29 20:52:56 on dist-test-slave-m9g1"
server_key: "c6067f9dae95649f8fe2eb987da337ad"
server_key_iv: "27d923bf614146138390bb116a6d521a"
server_key_version: "encryptionkey@0"
I20260929 20:52:56.468941 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d7cdcda32e8a421db84a88f5b0a49554"
format_stamp: "Formatted at 2026-09-29 20:52:56 on dist-test-slave-m9g1"
server_key: "c6067f9dae95649f8fe2eb987da337ad"
server_key_iv: "27d923bf614146138390bb116a6d521a"
server_key_version: "encryptionkey@0"
I20260929 20:52:56.473671 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260929 20:52:56.476836 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:56.478032 1023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:52:56.478163 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "d7cdcda32e8a421db84a88f5b0a49554"
format_stamp: "Formatted at 2026-09-29 20:52:56 on dist-test-slave-m9g1"
server_key: "c6067f9dae95649f8fe2eb987da337ad"
server_key_iv: "27d923bf614146138390bb116a6d521a"
server_key_version: "encryptionkey@0"
I20260929 20:52:56.478376 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:52:56.490923 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:52:56.494489 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:52:56.494741 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:52:56.495596 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:52:56.496799 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:52:56.496873 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:56.496986 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:52:56.497037 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:56.524554 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45715
I20260929 20:52:56.524588 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45715 every 8 connection(s)
I20260929 20:52:56.525746 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:52:56.533463 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 1023
I20260929 20:52:56.533641 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:52:56.534240 388 external_mini_cluster.cc:1428] Setting key ec2c55b784bf4eb5a5c8c1b257891d87
Sep 29 20:52:56 dist-test-slave-m9g1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:52:56.544323 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:56.527981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35431 (local address 127.0.97.62:35805)
0929 20:52:56.528355 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:56.528360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:56.529186 (+ 826us) server_negotiation.cc:407] Connection header received
0929 20:52:56.530955 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:56.530961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:56.531032 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:56.531147 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:56.532516 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:56.533216 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:56.534820 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:56.535334 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:56.535429 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:56.538658 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:56.538664 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:56.538668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:56.538719 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:56.541211 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:56.542090 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:56.542094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:56.542097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:56.542280 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:56.542723 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:56.542727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:56.542729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:56.543001 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:56.543138 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:56.543877 (+ 739us) server_negotiation.cc:299] Negotiation successful
0929 20:52:56.544066 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":206,"threads_started":1}
I20260929 20:52:56.545687 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:56.528429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35805 (local address 127.0.97.1:35431)
0929 20:52:56.529050 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:56.529086 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:56.530702 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:56.531353 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:56.531361 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:56.531845 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:56.532367 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:56.532382 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:56.533344 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:56.533347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:56.534670 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:56.534683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:56.535535 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:56.535542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:56.535798 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:56.536436 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:56.536458 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:56.538432 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:56.541335 (+ 2903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:56.541345 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:56.541363 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:56.541909 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:56.542381 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:56.542386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:56.542390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:56.542609 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:56.543148 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:56.543156 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:56.543542 (+ 386us) client_negotiation.cc:769] Sending connection context
0929 20:52:56.543816 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 20:52:56.544331 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":235,"threads_started":1}
I20260929 20:52:56.547283 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35805
I20260929 20:52:56.547581 1151 heartbeater.cc:464] Registering TS with master...
I20260929 20:52:56.548328 1151 heartbeater.cc:511] Master 127.0.97.62:35805 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:52:56.550388 962 ts_manager.cc:194] Registered new tserver with Master: d7cdcda32e8a421db84a88f5b0a49554 (127.0.97.1:45715)
I20260929 20:52:56.551759 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:52:56.571058 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35805
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:52:56.882648 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:52:56.882951 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:52:56.883051 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:52:56.890005 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:52:56.890105 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:52:56.890237 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:52:56.899637 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35805
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:52:56.900807 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:52:56.902012 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:52:56.913947 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:56.913980 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:56.914002 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:56.914817 1157 server_base.cc:1034] running on GCE node
I20260929 20:52:56.915320 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:52:56.916168 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:52:56.917342 1157 hybrid_clock.cc:648] HybridClock initialized: now 1790715176917295 us; error 55 us; skew 500 ppm
I20260929 20:52:56.917599 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:52:56 dist-test-slave-m9g1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:52:56.920500 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:52:56.921798 1157 webserver.cc:460] Webserver started at http://127.0.97.2:46689/ using document root <none> and password file <none>
I20260929 20:52:56.922530 1157 fs_manager.cc:359] Metadata directory not provided
I20260929 20:52:56.922636 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:52:56.922919 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:52:56.925722 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "7de655f3c4ec4da7a20470d837af86a4"
format_stamp: "Formatted at 2026-09-29 20:52:56 on dist-test-slave-m9g1"
server_key: "a5e8ac193ca3b5e9c07225a98e9c9ba8"
server_key_iv: "c03ab510479aac6d4c7d38bf8c46fba2"
server_key_version: "encryptionkey@0"
I20260929 20:52:56.926477 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7de655f3c4ec4da7a20470d837af86a4"
format_stamp: "Formatted at 2026-09-29 20:52:56 on dist-test-slave-m9g1"
server_key: "a5e8ac193ca3b5e9c07225a98e9c9ba8"
server_key_iv: "c03ab510479aac6d4c7d38bf8c46fba2"
server_key_version: "encryptionkey@0"
I20260929 20:52:56.931337 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:52:56.934597 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:56.935798 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:52:56.935945 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "7de655f3c4ec4da7a20470d837af86a4"
format_stamp: "Formatted at 2026-09-29 20:52:56 on dist-test-slave-m9g1"
server_key: "a5e8ac193ca3b5e9c07225a98e9c9ba8"
server_key_iv: "c03ab510479aac6d4c7d38bf8c46fba2"
server_key_version: "encryptionkey@0"
I20260929 20:52:56.936111 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:52:56.948534 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:52:56.952070 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:52:56.952303 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:52:56.953132 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:52:56.954370 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:52:56.954447 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:56.954555 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:52:56.954602 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:56.982810 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46213
I20260929 20:52:56.982832 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46213 every 8 connection(s)
I20260929 20:52:56.984211 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:52:56.986364 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 1157
I20260929 20:52:56.986557 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:52:56.987016 388 external_mini_cluster.cc:1428] Setting key 8fc2863316899fc3ea580f83a4b6b182
Sep 29 20:52:56 dist-test-slave-m9g1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 20:52:57.005400 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:56.986619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36125 (local address 127.0.97.62:35805)
0929 20:52:56.986795 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:56.986801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:56.987955 (+ 1154us) server_negotiation.cc:407] Connection header received
0929 20:52:56.990159 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:56.990165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:56.990279 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:56.990374 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:56.992240 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:56.993378 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:56.995670 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:56.996401 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:56.996516 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:57.000487 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:57.000492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:57.000495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:57.000526 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:57.002901 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:57.003517 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:57.003521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:57.003523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:57.003649 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:57.004017 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:57.004021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:57.004023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:57.004260 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:57.004363 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:57.004960 (+ 597us) server_negotiation.cc:299] Negotiation successful
0929 20:52:57.005142 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:52:57.006470 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:56.987069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35805 (local address 127.0.97.2:36125)
0929 20:52:56.987803 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:56.987851 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:56.989873 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:56.990629 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:56.990641 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:56.991297 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:56.992065 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:56.992089 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:56.993558 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:56.993564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:56.995507 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:56.995522 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:56.996618 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:56.996625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:56.996882 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:56.997644 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:56.997671 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:57.000273 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:57.003021 (+ 2748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:57.003030 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:57.003047 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:57.003427 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:57.003746 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:57.003750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:57.003752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:57.003920 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:57.004355 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:57.004361 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:57.004657 (+ 296us) client_negotiation.cc:769] Sending connection context
0929 20:52:57.004871 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 20:52:57.005355 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":289,"threads_started":1}
I20260929 20:52:57.007829 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35805
I20260929 20:52:57.008152 1285 heartbeater.cc:464] Registering TS with master...
I20260929 20:52:57.008939 1285 heartbeater.cc:511] Master 127.0.97.62:35805 requested a full tablet report, sending...
I20260929 20:52:57.010365 962 ts_manager.cc:194] Registered new tserver with Master: 7de655f3c4ec4da7a20470d837af86a4 (127.0.97.2:46213)
I20260929 20:52:57.011404 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:52:57.024391 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35805
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:52:57.324343 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:52:57.324602 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:52:57.324702 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:52:57.331664 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:52:57.331759 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:52:57.331897 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:52:57.341221 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35805
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:52:57.342463 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:52:57.343756 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:52:57.355737 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:57.355757 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:57.355830 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:57.356585 1290 server_base.cc:1034] running on GCE node
I20260929 20:52:57.357123 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:52:57.358038 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:52:57.359220 1290 hybrid_clock.cc:648] HybridClock initialized: now 1790715177359168 us; error 49 us; skew 500 ppm
I20260929 20:52:57.359513 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:52:57 dist-test-slave-m9g1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:52:57.362900 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:52:57.364565 1290 webserver.cc:460] Webserver started at http://127.0.97.3:36763/ using document root <none> and password file <none>
I20260929 20:52:57.365383 1290 fs_manager.cc:359] Metadata directory not provided
I20260929 20:52:57.365505 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:52:57.365846 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:52:57.369059 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "8938e7b99bc44e29845ae4f598b5de77"
format_stamp: "Formatted at 2026-09-29 20:52:57 on dist-test-slave-m9g1"
server_key: "7ae46c01d4904ef9ac300564658617e6"
server_key_iv: "bbe1d0fb0c8204ff1a2862d22ba165f2"
server_key_version: "encryptionkey@0"
I20260929 20:52:57.369757 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8938e7b99bc44e29845ae4f598b5de77"
format_stamp: "Formatted at 2026-09-29 20:52:57 on dist-test-slave-m9g1"
server_key: "7ae46c01d4904ef9ac300564658617e6"
server_key_iv: "bbe1d0fb0c8204ff1a2862d22ba165f2"
server_key_version: "encryptionkey@0"
I20260929 20:52:57.374698 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 20:52:57.377720 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:57.378944 1290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 20:52:57.379074 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "8938e7b99bc44e29845ae4f598b5de77"
format_stamp: "Formatted at 2026-09-29 20:52:57 on dist-test-slave-m9g1"
server_key: "7ae46c01d4904ef9ac300564658617e6"
server_key_iv: "bbe1d0fb0c8204ff1a2862d22ba165f2"
server_key_version: "encryptionkey@0"
I20260929 20:52:57.379231 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:52:57.398408 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:52:57.402436 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:52:57.402707 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:52:57.403578 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:52:57.404786 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:52:57.404860 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:57.404974 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:52:57.405025 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:57.432866 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43009
I20260929 20:52:57.432909 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43009 every 8 connection(s)
I20260929 20:52:57.434063 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:52:57.436977 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 1290
I20260929 20:52:57.437141 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:52:57.437518 388 external_mini_cluster.cc:1428] Setting key 50ce462bfeba64d3861a2f4e4fac3dcc
Sep 29 20:52:57 dist-test-slave-m9g1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:52:57.452584 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:57.436369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36999 (local address 127.0.97.62:35805)
0929 20:52:57.436534 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:57.436539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:57.437735 (+ 1196us) server_negotiation.cc:407] Connection header received
0929 20:52:57.439787 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:57.439793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:57.439873 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:57.439977 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:57.441478 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.442165 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.443648 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.444059 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.444149 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:57.447275 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:57.447280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:57.447284 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:57.447323 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:57.449760 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:57.450364 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:57.450368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:57.450370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:57.450479 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:57.450838 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:57.450842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:57.450845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:57.451140 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:57.451267 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:57.452045 (+ 778us) server_negotiation.cc:299] Negotiation successful
0929 20:52:57.452265 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:52:57.453457 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:57.436838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35805 (local address 127.0.97.3:36999)
0929 20:52:57.437577 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:57.437620 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:57.439523 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:57.440238 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:57.440246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:57.440785 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:57.441337 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:57.441354 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.442298 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.442301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:57.443527 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:57.443539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.444236 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.444240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:57.444436 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:57.444990 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:57.445014 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:57.447075 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:57.449858 (+ 2783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:57.449865 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:57.449877 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:57.450286 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:57.450551 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:57.450554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:57.450556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:57.450735 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:57.451261 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:57.451269 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:57.451638 (+ 369us) client_negotiation.cc:769] Sending connection context
0929 20:52:57.451900 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 20:52:57.452343 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":292,"threads_started":1}
I20260929 20:52:57.454620 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35805
I20260929 20:52:57.454861 1418 heartbeater.cc:464] Registering TS with master...
I20260929 20:52:57.455480 1418 heartbeater.cc:511] Master 127.0.97.62:35805 requested a full tablet report, sending...
I20260929 20:52:57.457093 962 ts_manager.cc:194] Registered new tserver with Master: 8938e7b99bc44e29845ae4f598b5de77 (127.0.97.3:43009)
I20260929 20:52:57.457826 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36999
I20260929 20:52:57.464567 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:52:57.481464 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:57.470294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57178 (local address 127.0.97.62:35805)
0929 20:52:57.470449 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:57.470453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:57.470903 (+ 450us) server_negotiation.cc:407] Connection header received
0929 20:52:57.471052 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:57.471056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:57.471112 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:57.471199 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:57.472510 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.473198 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.474454 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.474841 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.474943 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:57.477030 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:57.477035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:57.477038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:57.477067 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:57.479409 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:57.479942 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:57.479946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:57.479948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:57.480046 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:57.480297 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:57.480299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:57.480302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:57.480561 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:57.480734 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:57.480999 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 20:52:57.481172 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 20:52:57.487582 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:52:57.493939 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:52:57.519079 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:57.505994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59362 (local address 127.0.97.1:45715)
0929 20:52:57.507139 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:57.507148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:57.507573 (+ 425us) server_negotiation.cc:407] Connection header received
0929 20:52:57.507718 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:57.507726 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:57.507972 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:57.508159 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:52:57.509002 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.511208 (+ 2206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.515529 (+ 4321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.517051 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.517151 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:57.517358 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:57.518022 (+ 664us) server_negotiation.cc:299] Negotiation successful
0929 20:52:57.518246 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":588,"threads_started":1}
I20260929 20:52:57.520387 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:57.505521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45715 (local address 127.0.0.1:59362)
0929 20:52:57.507473 (+ 1952us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:57.507486 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:57.507591 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:57.508435 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:57.508442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:57.508473 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:52:57.508823 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:57.508835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.511948 (+ 3113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.511953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:57.515354 (+ 3401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:57.515369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.517546 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.517553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:57.517764 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:57.517813 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 20:52:57.519860 (+ 2047us) client_negotiation.cc:240] Negotiation successful
0929 20:52:57.520056 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1864,"thread_start_us":1746,"threads_started":1}
I20260929 20:52:57.524547 1086 tablet_service.cc:1467] Processing CreateTablet for tablet ca8df12fd8864f5e94df0b85b4cb5bfd (DEFAULT_TABLE table=test-table [id=c30b0d1b1da348a09a0dd22fb0433132]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:52:57.526104 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8df12fd8864f5e94df0b85b4cb5bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:52:57.528618 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:57.507641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46213 (local address 127.0.0.1:60216)
0929 20:52:57.520760 (+ 13119us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:57.520777 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:57.520966 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:57.521790 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:57.521793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:57.521811 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:52:57.522236 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:57.522245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.524000 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.524005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:57.525952 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:57.525966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.527888 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.527894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:57.528042 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:57.528071 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 20:52:57.528169 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 20:52:57.528285 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12979,"thread_start_us":12839,"threads_started":1}
I20260929 20:52:57.529065 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:57.505399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43009 (local address 127.0.0.1:42526)
0929 20:52:57.506142 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:57.506174 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:57.506353 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:57.521747 (+ 15394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:57.521752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:57.521776 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:52:57.522215 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:57.522228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.524708 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.524713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:57.526718 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:57.526731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.528402 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.528408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:57.528571 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:57.528602 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:52:57.528660 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 20:52:57.528781 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":404,"threads_started":1}
I20260929 20:52:57.529296 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:57.507968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60216 (local address 127.0.97.2:46213)
0929 20:52:57.520167 (+ 12199us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:57.520175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:57.520871 (+ 696us) server_negotiation.cc:407] Connection header received
0929 20:52:57.521138 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:57.521147 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:57.521395 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:57.521550 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:52:57.522395 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.523846 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.526125 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.527455 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.527548 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:57.527728 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:57.528867 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0929 20:52:57.529021 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12043,"thread_start_us":12219,"threads_started":1}
I20260929 20:52:57.530416 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:57.505545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42526 (local address 127.0.97.3:43009)
0929 20:52:57.521319 (+ 15774us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:57.521325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:57.521350 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:52:57.521422 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:57.521427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:57.521587 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:57.521762 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:52:57.522779 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.524598 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.526878 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.528192 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.529373 (+ 1181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:57.529871 (+ 498us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:57.529931 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 20:52:57.530079 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15644,"thread_start_us":15488,"threads_started":1}
I20260929 20:52:57.532436 1220 tablet_service.cc:1467] Processing CreateTablet for tablet ca8df12fd8864f5e94df0b85b4cb5bfd (DEFAULT_TABLE table=test-table [id=c30b0d1b1da348a09a0dd22fb0433132]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:52:57.533524 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8df12fd8864f5e94df0b85b4cb5bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:52:57.538040 1353 tablet_service.cc:1467] Processing CreateTablet for tablet ca8df12fd8864f5e94df0b85b4cb5bfd (DEFAULT_TABLE table=test-table [id=c30b0d1b1da348a09a0dd22fb0433132]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:52:57.539419 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8df12fd8864f5e94df0b85b4cb5bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:52:57.542327 1436 tablet_bootstrap.cc:492] T ca8df12fd8864f5e94df0b85b4cb5bfd P d7cdcda32e8a421db84a88f5b0a49554: Bootstrap starting.
I20260929 20:52:57.545579 1436 tablet_bootstrap.cc:654] T ca8df12fd8864f5e94df0b85b4cb5bfd P d7cdcda32e8a421db84a88f5b0a49554: Neither blocks nor log segments found. Creating new log.
I20260929 20:52:57.546730 1436 log.cc:824] T ca8df12fd8864f5e94df0b85b4cb5bfd P d7cdcda32e8a421db84a88f5b0a49554: Log is configured to *not* fsync() on all Append() calls
I20260929 20:52:57.547254 1437 tablet_bootstrap.cc:492] T ca8df12fd8864f5e94df0b85b4cb5bfd P 7de655f3c4ec4da7a20470d837af86a4: Bootstrap starting.
I20260929 20:52:57.550487 1436 tablet_bootstrap.cc:492] T ca8df12fd8864f5e94df0b85b4cb5bfd P d7cdcda32e8a421db84a88f5b0a49554: No bootstrap required, opened a new log
I20260929 20:52:57.550747 1436 ts_tablet_manager.cc:1397] T ca8df12fd8864f5e94df0b85b4cb5bfd P d7cdcda32e8a421db84a88f5b0a49554: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 20:52:57.551222 1437 tablet_bootstrap.cc:654] T ca8df12fd8864f5e94df0b85b4cb5bfd P 7de655f3c4ec4da7a20470d837af86a4: Neither blocks nor log segments found. Creating new log.
I20260929 20:52:57.552343 1437 log.cc:824] T ca8df12fd8864f5e94df0b85b4cb5bfd P 7de655f3c4ec4da7a20470d837af86a4: Log is configured to *not* fsync() on all Append() calls
I20260929 20:52:57.552520 1439 tablet_bootstrap.cc:492] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77: Bootstrap starting.
I20260929 20:52:57.555058 1437 tablet_bootstrap.cc:492] T ca8df12fd8864f5e94df0b85b4cb5bfd P 7de655f3c4ec4da7a20470d837af86a4: No bootstrap required, opened a new log
I20260929 20:52:57.555053 1151 heartbeater.cc:503] Master 127.0.97.62:35805 was elected leader, sending a full tablet report...
I20260929 20:52:57.555308 1437 ts_tablet_manager.cc:1397] T ca8df12fd8864f5e94df0b85b4cb5bfd P 7de655f3c4ec4da7a20470d837af86a4: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 20:52:57.555847 1439 tablet_bootstrap.cc:654] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77: Neither blocks nor log segments found. Creating new log.
I20260929 20:52:57.556957 1439 log.cc:824] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77: Log is configured to *not* fsync() on all Append() calls
I20260929 20:52:57.560608 1436 raft_consensus.cc:348] T ca8df12fd8864f5e94df0b85b4cb5bfd P d7cdcda32e8a421db84a88f5b0a49554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8938e7b99bc44e29845ae4f598b5de77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43009 } } peers { permanent_uuid: "d7cdcda32e8a421db84a88f5b0a49554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45715 } } peers { permanent_uuid: "7de655f3c4ec4da7a20470d837af86a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46213 } }
I20260929 20:52:57.561017 1436 raft_consensus.cc:374] T ca8df12fd8864f5e94df0b85b4cb5bfd P d7cdcda32e8a421db84a88f5b0a49554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:52:57.561108 1436 raft_consensus.cc:729] T ca8df12fd8864f5e94df0b85b4cb5bfd P d7cdcda32e8a421db84a88f5b0a49554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cdcda32e8a421db84a88f5b0a49554, State: Initialized, Role: FOLLOWER
I20260929 20:52:57.561268 1439 tablet_bootstrap.cc:492] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77: No bootstrap required, opened a new log
I20260929 20:52:57.561496 1439 ts_tablet_manager.cc:1397] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260929 20:52:57.562255 1436 consensus_queue.cc:260] T ca8df12fd8864f5e94df0b85b4cb5bfd P d7cdcda32e8a421db84a88f5b0a49554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8938e7b99bc44e29845ae4f598b5de77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43009 } } peers { permanent_uuid: "d7cdcda32e8a421db84a88f5b0a49554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45715 } } peers { permanent_uuid: "7de655f3c4ec4da7a20470d837af86a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46213 } }
I20260929 20:52:57.563822 1436 ts_tablet_manager.cc:1428] T ca8df12fd8864f5e94df0b85b4cb5bfd P d7cdcda32e8a421db84a88f5b0a49554: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260929 20:52:57.565601 1437 raft_consensus.cc:348] T ca8df12fd8864f5e94df0b85b4cb5bfd P 7de655f3c4ec4da7a20470d837af86a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8938e7b99bc44e29845ae4f598b5de77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43009 } } peers { permanent_uuid: "d7cdcda32e8a421db84a88f5b0a49554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45715 } } peers { permanent_uuid: "7de655f3c4ec4da7a20470d837af86a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46213 } }
I20260929 20:52:57.566004 1437 raft_consensus.cc:374] T ca8df12fd8864f5e94df0b85b4cb5bfd P 7de655f3c4ec4da7a20470d837af86a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:52:57.566093 1437 raft_consensus.cc:729] T ca8df12fd8864f5e94df0b85b4cb5bfd P 7de655f3c4ec4da7a20470d837af86a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de655f3c4ec4da7a20470d837af86a4, State: Initialized, Role: FOLLOWER
I20260929 20:52:57.566655 1437 consensus_queue.cc:260] T ca8df12fd8864f5e94df0b85b4cb5bfd P 7de655f3c4ec4da7a20470d837af86a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8938e7b99bc44e29845ae4f598b5de77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43009 } } peers { permanent_uuid: "d7cdcda32e8a421db84a88f5b0a49554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45715 } } peers { permanent_uuid: "7de655f3c4ec4da7a20470d837af86a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46213 } }
I20260929 20:52:57.567602 1285 heartbeater.cc:503] Master 127.0.97.62:35805 was elected leader, sending a full tablet report...
I20260929 20:52:57.569334 1437 ts_tablet_manager.cc:1428] T ca8df12fd8864f5e94df0b85b4cb5bfd P 7de655f3c4ec4da7a20470d837af86a4: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260929 20:52:57.569746 1439 raft_consensus.cc:348] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8938e7b99bc44e29845ae4f598b5de77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43009 } } peers { permanent_uuid: "d7cdcda32e8a421db84a88f5b0a49554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45715 } } peers { permanent_uuid: "7de655f3c4ec4da7a20470d837af86a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46213 } }
I20260929 20:52:57.570034 1439 raft_consensus.cc:374] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:52:57.570122 1439 raft_consensus.cc:729] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8938e7b99bc44e29845ae4f598b5de77, State: Initialized, Role: FOLLOWER
I20260929 20:52:57.570657 1439 consensus_queue.cc:260] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8938e7b99bc44e29845ae4f598b5de77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43009 } } peers { permanent_uuid: "d7cdcda32e8a421db84a88f5b0a49554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45715 } } peers { permanent_uuid: "7de655f3c4ec4da7a20470d837af86a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46213 } }
I20260929 20:52:57.571554 1418 heartbeater.cc:503] Master 127.0.97.62:35805 was elected leader, sending a full tablet report...
I20260929 20:52:57.572053 1439 ts_tablet_manager.cc:1428] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
W20260929 20:52:57.685555 1419 tablet.cc:2369] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:52:57.736052 1286 tablet.cc:2369] T ca8df12fd8864f5e94df0b85b4cb5bfd P 7de655f3c4ec4da7a20470d837af86a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:52:57.742372 1444 raft_consensus.cc:482] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:52:57.742626 1444 raft_consensus.cc:504] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8938e7b99bc44e29845ae4f598b5de77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43009 } } peers { permanent_uuid: "d7cdcda32e8a421db84a88f5b0a49554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45715 } } peers { permanent_uuid: "7de655f3c4ec4da7a20470d837af86a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46213 } }
I20260929 20:52:57.744385 1444 leader_election.cc:288] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7cdcda32e8a421db84a88f5b0a49554 (127.0.97.1:45715), 7de655f3c4ec4da7a20470d837af86a4 (127.0.97.2:46213)
I20260929 20:52:57.750530 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:57.744730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45715 (local address 127.0.97.3:38105)
0929 20:52:57.744933 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:57.744952 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:57.745081 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:57.745536 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:57.745541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:57.745591 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:52:57.745782 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:57.745793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.746961 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.746964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:57.748718 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:57.748732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.749990 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.749994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:57.750120 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:57.750143 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:52:57.750250 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 20:52:57.750340 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 20:52:57.750641 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:57.744731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38105 (local address 127.0.97.1:45715)
0929 20:52:57.744935 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:57.744941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:57.745020 (+ 79us) server_negotiation.cc:407] Connection header received
0929 20:52:57.745244 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:57.745250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:57.745324 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:57.745416 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:52:57.745914 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.746820 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.748851 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.749845 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.749966 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:57.750072 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:57.750278 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 20:52:57.750404 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:52:57.751574 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:57.745100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46213 (local address 127.0.97.3:49073)
0929 20:52:57.745784 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:57.745799 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:57.745910 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:57.746207 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:57.746211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:57.746233 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:52:57.746477 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:57.746485 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.747508 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.747513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:57.749222 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:57.749232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.750915 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.750921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:57.751052 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:57.751090 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 20:52:57.751179 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 20:52:57.751290 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":358,"threads_started":1}
I20260929 20:52:57.751537 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8df12fd8864f5e94df0b85b4cb5bfd" candidate_uuid: "8938e7b99bc44e29845ae4f598b5de77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cdcda32e8a421db84a88f5b0a49554" is_pre_election: true
I20260929 20:52:57.751930 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:57.745265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49073 (local address 127.0.97.2:46213)
0929 20:52:57.745442 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:57.745448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:57.745884 (+ 436us) server_negotiation.cc:407] Connection header received
0929 20:52:57.746016 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:57.746021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:57.746090 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:57.746178 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:52:57.746601 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.747381 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.749346 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.750464 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.750593 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:57.750675 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:57.751209 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 20:52:57.751293 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 20:52:57.752112 1106 raft_consensus.cc:2454] T ca8df12fd8864f5e94df0b85b4cb5bfd P d7cdcda32e8a421db84a88f5b0a49554 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8938e7b99bc44e29845ae4f598b5de77 in term 0.
I20260929 20:52:57.752707 1307 leader_election.cc:302] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8938e7b99bc44e29845ae4f598b5de77, d7cdcda32e8a421db84a88f5b0a49554; no voters:
I20260929 20:52:57.752636 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8df12fd8864f5e94df0b85b4cb5bfd" candidate_uuid: "8938e7b99bc44e29845ae4f598b5de77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7de655f3c4ec4da7a20470d837af86a4" is_pre_election: true
I20260929 20:52:57.753034 1444 raft_consensus.cc:2781] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:52:57.753067 1240 raft_consensus.cc:2454] T ca8df12fd8864f5e94df0b85b4cb5bfd P 7de655f3c4ec4da7a20470d837af86a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8938e7b99bc44e29845ae4f598b5de77 in term 0.
I20260929 20:52:57.753182 1444 raft_consensus.cc:482] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:52:57.753274 1444 raft_consensus.cc:3037] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:52:57.755394 1444 raft_consensus.cc:504] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8938e7b99bc44e29845ae4f598b5de77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43009 } } peers { permanent_uuid: "d7cdcda32e8a421db84a88f5b0a49554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45715 } } peers { permanent_uuid: "7de655f3c4ec4da7a20470d837af86a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46213 } }
I20260929 20:52:57.755860 1444 leader_election.cc:288] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77 [CANDIDATE]: Term 1 election: Requested vote from peers d7cdcda32e8a421db84a88f5b0a49554 (127.0.97.1:45715), 7de655f3c4ec4da7a20470d837af86a4 (127.0.97.2:46213)
I20260929 20:52:57.756219 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8df12fd8864f5e94df0b85b4cb5bfd" candidate_uuid: "8938e7b99bc44e29845ae4f598b5de77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cdcda32e8a421db84a88f5b0a49554"
I20260929 20:52:57.756280 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8df12fd8864f5e94df0b85b4cb5bfd" candidate_uuid: "8938e7b99bc44e29845ae4f598b5de77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7de655f3c4ec4da7a20470d837af86a4"
I20260929 20:52:57.756367 1106 raft_consensus.cc:3037] T ca8df12fd8864f5e94df0b85b4cb5bfd P d7cdcda32e8a421db84a88f5b0a49554 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:52:57.756453 1240 raft_consensus.cc:3037] T ca8df12fd8864f5e94df0b85b4cb5bfd P 7de655f3c4ec4da7a20470d837af86a4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:52:57.758378 1106 raft_consensus.cc:2454] T ca8df12fd8864f5e94df0b85b4cb5bfd P d7cdcda32e8a421db84a88f5b0a49554 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8938e7b99bc44e29845ae4f598b5de77 in term 1.
I20260929 20:52:57.758423 1240 raft_consensus.cc:2454] T ca8df12fd8864f5e94df0b85b4cb5bfd P 7de655f3c4ec4da7a20470d837af86a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8938e7b99bc44e29845ae4f598b5de77 in term 1.
I20260929 20:52:57.758692 1307 leader_election.cc:302] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8938e7b99bc44e29845ae4f598b5de77, d7cdcda32e8a421db84a88f5b0a49554; no voters:
I20260929 20:52:57.758916 1444 raft_consensus.cc:2781] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:52:57.759316 1444 raft_consensus.cc:686] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77 [term 1 LEADER]: Becoming Leader. State: Replica: 8938e7b99bc44e29845ae4f598b5de77, State: Running, Role: LEADER
I20260929 20:52:57.759766 1444 consensus_queue.cc:237] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8938e7b99bc44e29845ae4f598b5de77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43009 } } peers { permanent_uuid: "d7cdcda32e8a421db84a88f5b0a49554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45715 } } peers { permanent_uuid: "7de655f3c4ec4da7a20470d837af86a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46213 } }
I20260929 20:52:57.764423 961 catalog_manager.cc:5495] T ca8df12fd8864f5e94df0b85b4cb5bfd P 8938e7b99bc44e29845ae4f598b5de77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8938e7b99bc44e29845ae4f598b5de77 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8938e7b99bc44e29845ae4f598b5de77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8938e7b99bc44e29845ae4f598b5de77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7cdcda32e8a421db84a88f5b0a49554" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7de655f3c4ec4da7a20470d837af86a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46213 } health_report { overall_health: UNKNOWN } } }
W20260929 20:52:57.779795 1152 tablet.cc:2369] T ca8df12fd8864f5e94df0b85b4cb5bfd P d7cdcda32e8a421db84a88f5b0a49554: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 20:52:57 dist-test-slave-m9g1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:52:57.812906 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:57.799725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59374 (local address 127.0.97.1:45715)
0929 20:52:57.799880 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:57.799884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:57.800360 (+ 476us) server_negotiation.cc:407] Connection header received
0929 20:52:57.800483 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:57.800487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:57.800564 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:57.800648 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:57.801950 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.802714 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.803853 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.804236 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.804321 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:57.807914 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:57.807927 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:57.807943 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:57.807972 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:57.810865 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:57.811407 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:57.811413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:57.811415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:57.811516 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:57.811762 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:57.811764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:57.811766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:57.812092 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:57.812290 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:57.812475 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 20:52:57.812616 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 20:52:57 dist-test-slave-m9g1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715177, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:52:57 dist-test-slave-m9g1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715177, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:52:57.853281 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:57.840501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59380 (local address 127.0.97.1:45715)
0929 20:52:57.840702 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:57.840709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:57.840946 (+ 237us) server_negotiation.cc:407] Connection header received
0929 20:52:57.841126 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:57.841130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:57.841188 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:57.841268 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:57.842529 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.843247 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.844492 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.844889 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.844996 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:57.848648 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:57.848653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:57.848656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:57.848690 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:57.850978 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:57.851543 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:57.851547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:57.851549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:57.851669 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:57.851998 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:57.852003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:57.852006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:57.852267 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:57.852480 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:57.852765 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 20:52:57.852963 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 20:52:57 dist-test-slave-m9g1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:52:57 dist-test-slave-m9g1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:52:57.892326 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:57.880003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.0.97.1:45715)
0929 20:52:57.880188 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:57.880195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:57.880472 (+ 277us) server_negotiation.cc:407] Connection header received
0929 20:52:57.880640 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:57.880644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:57.880704 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:57.880781 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:57.882089 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.882832 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.883973 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.884392 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.884507 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:57.887904 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:57.887909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:57.887912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:57.887941 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:57.890028 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:57.890620 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:57.890625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:57.890629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:57.890736 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:57.891049 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:57.891055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:57.891058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:57.891329 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:57.891565 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:57.891805 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 20:52:57.892013 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 20:52:57.893955 1066 tablet_service.cc:2908] Rejecting scan request for tablet ca8df12fd8864f5e94df0b85b4cb5bfd: Uninitialized: clean time has not yet been initialized
I20260929 20:52:57.896675 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 1023
I20260929 20:52:57.907385 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:57.897452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59396 (local address 127.0.97.1:45715)
0929 20:52:57.897590 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:57.897594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:57.897884 (+ 290us) server_negotiation.cc:407] Connection header received
0929 20:52:57.898029 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:57.898033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:57.898097 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:57.898166 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:57.899151 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.899782 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.900839 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:57.901604 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:57.901700 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:57.903533 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:57.903537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:57.903540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:57.903566 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:57.905551 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:57.906066 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:57.906070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:57.906072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:57.906168 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:57.906427 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:57.906430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:57.906432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:57.906664 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:57.906848 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:57.907018 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 20:52:57.907171 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 20:52:57.907922 1146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:52:58.000082 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 1023
I20260929 20:52:58.013207 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 1157
Sep 29 20:52:58 dist-test-slave-m9g1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:52:58.025875 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:58.013948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60232 (local address 127.0.97.2:46213)
0929 20:52:58.014092 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:58.014096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:58.014112 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:52:58.014178 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:58.014199 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:58.014294 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:58.014371 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:58.015349 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:58.016072 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:58.017192 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:58.017577 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:58.017657 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:58.021316 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:58.021329 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:58.021345 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:58.021375 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:58.023865 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:58.024387 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:58.024402 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:58.024404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:58.024509 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:58.024771 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:58.024773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:58.024775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:58.025099 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:58.025286 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:58.025495 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 20:52:58.025635 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 20:52:58.026540 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:52:58.110263 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 1157
I20260929 20:52:58.123493 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 1290
Sep 29 20:52:58 dist-test-slave-m9g1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:52:58.136474 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:58.124175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42536 (local address 127.0.97.3:43009)
0929 20:52:58.124563 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:58.124567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:58.124583 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:52:58.124637 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:58.124641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:58.124697 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:58.124793 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:58.125834 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:58.126536 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:58.127670 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:58.128049 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:58.128136 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:58.131745 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:58.131763 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:58.131780 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:58.131811 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:58.134232 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:58.134842 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:58.134848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:58.134850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:58.134982 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:58.135237 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:58.135240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:58.135242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:58.135592 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:58.135799 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:58.135966 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 20:52:58.136117 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":204,"threads_started":1}
I20260929 20:52:58.137096 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:52:58.221475 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 1290
I20260929 20:52:58.234055 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 930
I20260929 20:52:58.235117 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:52:58.338495 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 930
2026-09-29T20:52:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3878 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:52:58 dist-test-slave-m9g1 krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:52:58 dist-test-slave-m9g1 krb5kdc[1466](info): set up 2 sockets
Sep 29 20:52:58 dist-test-slave-m9g1 krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260929 20:52:59.333926 388 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.004s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:52:59 dist-test-slave-m9g1 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:52:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:52:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:52:59.497210 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39989
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:52:59.805433 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:52:59.805738 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:52:59.805869 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:52:59.813054 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:52:59.813165 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 20:52:59.813230 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:52:59.813315 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:52:59.813377 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:52:59.813468 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:52:59.823329 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39989
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:52:59.824616 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:52:59.825866 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:52:59.837141 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:59.837070 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:59.837117 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:59.838348 1479 server_base.cc:1034] running on GCE node
I20260929 20:52:59.839128 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:52:59.840561 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:52:59.841784 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790715179841719 us; error 39 us; skew 500 ppm
I20260929 20:52:59.842093 1479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:52:59 dist-test-slave-m9g1 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:52:59.844993 1479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:52:59.846495 1479 webserver.cc:460] Webserver started at http://127.0.97.62:43465/ using document root <none> and password file <none>
I20260929 20:52:59.847187 1479 fs_manager.cc:359] Metadata directory not provided
I20260929 20:52:59.847285 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:52:59.847602 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:52:59.850631 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "d6c89bced60149259508590bf327aa0d"
format_stamp: "Formatted at 2026-09-29 20:52:59 on dist-test-slave-m9g1"
server_key: "f4a2f4d35bbf19ab288e187221728e85"
server_key_iv: "516fead5822f3b896c4a1bfccb48a359"
server_key_version: "encryptionkey@0"
I20260929 20:52:59.851399 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "d6c89bced60149259508590bf327aa0d"
format_stamp: "Formatted at 2026-09-29 20:52:59 on dist-test-slave-m9g1"
server_key: "f4a2f4d35bbf19ab288e187221728e85"
server_key_iv: "516fead5822f3b896c4a1bfccb48a359"
server_key_version: "encryptionkey@0"
I20260929 20:52:59.856407 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 20:52:59.859755 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:59.861063 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:52:59.861218 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "d6c89bced60149259508590bf327aa0d"
format_stamp: "Formatted at 2026-09-29 20:52:59 on dist-test-slave-m9g1"
server_key: "f4a2f4d35bbf19ab288e187221728e85"
server_key_iv: "516fead5822f3b896c4a1bfccb48a359"
server_key_version: "encryptionkey@0"
I20260929 20:52:59.861413 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:52:59.901758 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:52:59.905591 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:52:59.905836 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:52:59.925189 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39989
I20260929 20:52:59.925206 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39989 every 8 connection(s)
I20260929 20:52:59.926645 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:52:59.930508 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:52:59.932397 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 1479
I20260929 20:52:59.932622 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:52:59.933337 388 external_mini_cluster.cc:1428] Setting key de88def97195338102a432580b58a4af
I20260929 20:52:59.943109 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d: Bootstrap starting.
I20260929 20:52:59.946929 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d: Neither blocks nor log segments found. Creating new log.
Sep 29 20:52:59 dist-test-slave-m9g1 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:52:59.948108 1556 log.cc:824] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:52:59.951500 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d: No bootstrap required, opened a new log
I20260929 20:52:59.956269 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:59.935496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.0.97.62:39989)
0929 20:52:59.936301 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:59.936314 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:59.936360 (+ 46us) server_negotiation.cc:407] Connection header received
0929 20:52:59.938047 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:59.938070 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:59.938527 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:59.938856 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:59.940526 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:59.941842 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:59.943513 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:59.944189 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:59.944320 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:59.949019 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:59.949036 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:59.949063 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:59.949111 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:59.952196 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:59.953066 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:59.953072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:59.953079 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:59.953244 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:59.953609 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:59.953613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:59.953615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:59.954141 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:59.954401 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:59.954822 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 20:52:59.955206 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":361,"threads_started":1}
I20260929 20:52:59.960163 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c89bced60149259508590bf327aa0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39989 } }
I20260929 20:52:59.960495 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:52:59.960582 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c89bced60149259508590bf327aa0d, State: Initialized, Role: FOLLOWER
I20260929 20:52:59.961144 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c89bced60149259508590bf327aa0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39989 } }
I20260929 20:52:59.961380 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:52:59.961503 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:52:59.961648 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:52:59.963752 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c89bced60149259508590bf327aa0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39989 } }
I20260929 20:52:59.964222 1556 leader_election.cc:302] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6c89bced60149259508590bf327aa0d; no voters:
I20260929 20:52:59.964790 1556 leader_election.cc:288] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:52:59.964928 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:52:59.966040 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d [term 1 LEADER]: Becoming Leader. State: Replica: d6c89bced60149259508590bf327aa0d, State: Running, Role: LEADER
I20260929 20:52:59.966562 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c89bced60149259508590bf327aa0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39989 } }
I20260929 20:52:59.967043 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:52:59.969179 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6c89bced60149259508590bf327aa0d. Latest consensus state: current_term: 1 leader_uuid: "d6c89bced60149259508590bf327aa0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c89bced60149259508590bf327aa0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39989 } } }
I20260929 20:52:59.969483 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d [sys.catalog]: This master's current role is: LEADER
I20260929 20:52:59.969957 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6c89bced60149259508590bf327aa0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c89bced60149259508590bf327aa0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39989 } } }
I20260929 20:52:59.970305 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d [sys.catalog]: This master's current role is: LEADER
I20260929 20:52:59.971095 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:52:59.975654 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:52:59.983903 1570 catalog_manager.cc:1347] Generated new cluster ID: f4c58f77aaa247c19fd59c518c1d856c
I20260929 20:52:59.984031 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:52:59.998433 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:52:59.999773 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:53:00.009291 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d: Generated new TSK 0
I20260929 20:53:00.010244 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:53:00.061300 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39989
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:00.385831 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:00.386114 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:00.386262 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:00.393406 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:00.393517 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:00.393628 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:53:00.393733 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:53:00.403494 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:00.404783 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:00.406153 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:00.418556 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:00.418617 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:00.418597 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:00.419235 1582 server_base.cc:1034] running on GCE node
I20260929 20:53:00.419941 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:00.420799 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:00.421998 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790715180421929 us; error 44 us; skew 500 ppm
I20260929 20:53:00.422323 1582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:00 dist-test-slave-m9g1 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:00.425194 1582 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:53:00.426576 1582 webserver.cc:460] Webserver started at http://127.0.97.1:45577/ using document root <none> and password file <none>
I20260929 20:53:00.427258 1582 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:00.427351 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:00.427641 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:00.430646 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "4664b98e374042c3970dee6322790e54"
format_stamp: "Formatted at 2026-09-29 20:53:00 on dist-test-slave-m9g1"
server_key: "16afc54100246ccc2ec85c83bc17aea9"
server_key_iv: "86a787de06334ae7747022eaa6956a03"
server_key_version: "encryptionkey@0"
I20260929 20:53:00.431416 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4664b98e374042c3970dee6322790e54"
format_stamp: "Formatted at 2026-09-29 20:53:00 on dist-test-slave-m9g1"
server_key: "16afc54100246ccc2ec85c83bc17aea9"
server_key_iv: "86a787de06334ae7747022eaa6956a03"
server_key_version: "encryptionkey@0"
I20260929 20:53:00.436419 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 20:53:00.439900 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:00.441251 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:53:00.441398 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "4664b98e374042c3970dee6322790e54"
format_stamp: "Formatted at 2026-09-29 20:53:00 on dist-test-slave-m9g1"
server_key: "16afc54100246ccc2ec85c83bc17aea9"
server_key_iv: "86a787de06334ae7747022eaa6956a03"
server_key_version: "encryptionkey@0"
I20260929 20:53:00.441601 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:00.461145 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:00.465188 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:00.465451 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:00.468317 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:00.468402 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:00.468525 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:00.468580 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:53:00 dist-test-slave-m9g1 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:00.493467 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:00.469591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39989 (local address 127.0.97.1:45005)
0929 20:53:00.470550 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:00.470605 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:00.473150 (+ 2545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:00.473932 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:00.473944 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:00.474700 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:00.475579 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:00.475607 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:00.476772 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:00.476778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:00.479490 (+ 2712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:00.479508 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:00.480822 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:00.480830 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:00.481116 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:00.481878 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:00.481910 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:00.485305 (+ 3395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:00.488392 (+ 3087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:00.488403 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:00.488425 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:00.489041 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:00.489505 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:00.489513 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:00.489516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:00.489832 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:00.490400 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:00.490408 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:00.490929 (+ 521us) client_negotiation.cc:769] Sending connection context
0929 20:53:00.491291 (+ 362us) client_negotiation.cc:240] Negotiation successful
0929 20:53:00.491874 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":391,"threads_started":1}
I20260929 20:53:00.494812 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:00.468871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45005 (local address 127.0.97.62:39989)
0929 20:53:00.471357 (+ 2486us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:00.471364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:00.471386 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:53:00.473474 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:00.473479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:00.473548 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:00.473636 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:00.475779 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:00.476617 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:00.479686 (+ 3069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:00.480342 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:00.480446 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:00.485564 (+ 5118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:00.485569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:00.485573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:00.485607 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:00.488239 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:00.489208 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:00.489212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:00.489214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:00.489368 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:00.489947 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:00.489952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:00.489954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:00.490278 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:00.490406 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:00.494227 (+ 3821us) server_negotiation.cc:299] Negotiation successful
0929 20:53:00.494469 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2377,"thread_start_us":2283,"threads_started":1}
I20260929 20:53:00.514768 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:00.501103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33133 (local address 127.0.97.62:39989)
0929 20:53:00.501255 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:00.501260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:00.501694 (+ 434us) server_negotiation.cc:407] Connection header received
0929 20:53:00.501875 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:00.501879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:00.501941 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:00.502032 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:00.503563 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:00.504408 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:00.506587 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:00.507138 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:00.507265 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:00.509738 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:00.509743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:00.509746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:00.509777 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:00.512098 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:00.512868 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:00.512873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:00.512876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:00.513022 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:00.513415 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:00.513420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:00.513423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:00.513692 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:00.513817 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:00.514224 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 20:53:00.514426 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 20:53:00.514858 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:00.500815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39989 (local address 127.0.97.1:33133)
0929 20:53:00.501593 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:00.501608 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:00.501750 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:00.502084 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:00.502089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:00.502808 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:00.503396 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:00.503411 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:00.504566 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:00.504573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:00.506427 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:00.506445 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:00.507512 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:00.507519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:00.507704 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:00.508413 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:00.508430 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:00.509532 (+ 1102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:00.512239 (+ 2707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:00.512244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:00.512249 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:00.512727 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:00.513152 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:00.513168 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:00.513172 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:00.513294 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:00.513798 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:00.513802 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:00.513974 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 20:53:00.514086 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 20:53:00.514405 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":539,"threads_started":1}
I20260929 20:53:00.518410 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41841
I20260929 20:53:00.518441 1717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41841 every 8 connection(s)
I20260929 20:53:00.519718 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:53:00.520140 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 1582
I20260929 20:53:00.520366 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:53:00.521121 388 external_mini_cluster.cc:1428] Setting key 3c85ef6b2a0e46e604e276a9963d8483
I20260929 20:53:00.521735 1718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39989
I20260929 20:53:00.522012 1718 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:00.522650 1718 heartbeater.cc:511] Master 127.0.97.62:39989 requested a full tablet report, sending...
I20260929 20:53:00.524528 1511 ts_manager.cc:194] Registered new tserver with Master: 4664b98e374042c3970dee6322790e54 (127.0.97.1:41841)
I20260929 20:53:00.525939 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45005
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:53:00.557114 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39989
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:00.871978 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:00.872272 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:00.872395 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:00.879638 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:00.879750 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:00.879868 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:53:00.879968 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:53:00.889734 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:00.891047 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:00.892370 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:00.905206 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:00.905172 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:00.905673 1722 server_base.cc:1034] running on GCE node
W20260929 20:53:00.905242 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:00.906440 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:00.907353 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:00.908566 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790715180908498 us; error 44 us; skew 500 ppm
I20260929 20:53:00.908877 1722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:00 dist-test-slave-m9g1 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:00.911891 1722 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:53:00.913249 1722 webserver.cc:460] Webserver started at http://127.0.97.2:32885/ using document root <none> and password file <none>
I20260929 20:53:00.913923 1722 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:00.914026 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:00.914372 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:00.917227 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "3af7d71f8f7340c3b27d110f32891003"
format_stamp: "Formatted at 2026-09-29 20:53:00 on dist-test-slave-m9g1"
server_key: "c372ad8814e737e965f20a4de149e72b"
server_key_iv: "8326e012a9c71dd006c088ec4ee18c1a"
server_key_version: "encryptionkey@0"
I20260929 20:53:00.917960 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3af7d71f8f7340c3b27d110f32891003"
format_stamp: "Formatted at 2026-09-29 20:53:00 on dist-test-slave-m9g1"
server_key: "c372ad8814e737e965f20a4de149e72b"
server_key_iv: "8326e012a9c71dd006c088ec4ee18c1a"
server_key_version: "encryptionkey@0"
I20260929 20:53:00.923034 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:53:00.926630 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:00.928045 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:53:00.928186 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "3af7d71f8f7340c3b27d110f32891003"
format_stamp: "Formatted at 2026-09-29 20:53:00 on dist-test-slave-m9g1"
server_key: "c372ad8814e737e965f20a4de149e72b"
server_key_iv: "8326e012a9c71dd006c088ec4ee18c1a"
server_key_version: "encryptionkey@0"
I20260929 20:53:00.928377 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:00.949785 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:00.954061 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:00.954438 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:00.957067 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:00.957218 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:00.957341 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:00.957397 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:53:00 dist-test-slave-m9g1 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:00.980954 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:00.957883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60823 (local address 127.0.97.62:39989)
0929 20:53:00.958081 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:00.958088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:00.959595 (+ 1507us) server_negotiation.cc:407] Connection header received
0929 20:53:00.962640 (+ 3045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:00.962647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:00.962750 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:00.962863 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:00.965339 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:00.966574 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:00.969143 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:00.969697 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:00.969818 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:00.974962 (+ 5144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:00.974967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:00.974971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:00.975004 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:00.977397 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:00.978356 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:00.978360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:00.978362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:00.978503 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:00.979033 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:00.979037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:00.979040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:00.979322 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:00.979450 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:00.980464 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0929 20:53:00.980710 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:53:00.982872 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:00.958534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39989 (local address 127.0.97.2:60823)
0929 20:53:00.959408 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:00.959463 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:00.962254 (+ 2791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:00.963218 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:00.963231 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:00.964099 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:00.965057 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:00.965087 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:00.966732 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:00.966736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:00.968929 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:00.968951 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:00.970201 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:00.970210 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:00.970515 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:00.971432 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:00.971480 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:00.974677 (+ 3197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:00.977533 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:00.977544 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:00.977564 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:00.978240 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:00.978598 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:00.978603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:00.978607 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:00.978889 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:00.979455 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:00.979464 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:00.979974 (+ 510us) client_negotiation.cc:769] Sending connection context
0929 20:53:00.980322 (+ 348us) client_negotiation.cc:240] Negotiation successful
0929 20:53:00.980973 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":342,"threads_started":1}
I20260929 20:53:00.999034 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38095
I20260929 20:53:00.999110 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38095 every 8 connection(s)
I20260929 20:53:01.000634 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:53:01.002888 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 1722
I20260929 20:53:01.003079 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:53:01.003615 388 external_mini_cluster.cc:1428] Setting key e95887a23ecd1dc34fd82067cb63cd01
I20260929 20:53:01.004607 1857 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39989
I20260929 20:53:01.004854 1857 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:01.005661 1857 heartbeater.cc:511] Master 127.0.97.62:39989 requested a full tablet report, sending...
I20260929 20:53:01.007167 1511 ts_manager.cc:194] Registered new tserver with Master: 3af7d71f8f7340c3b27d110f32891003 (127.0.97.2:38095)
I20260929 20:53:01.008183 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60823
I20260929 20:53:01.010598 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:00.993477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58029 (local address 127.0.97.62:39989)
0929 20:53:00.993654 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:00.993660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:00.994493 (+ 833us) server_negotiation.cc:407] Connection header received
0929 20:53:00.994709 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:00.994715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:00.994822 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:00.994939 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:00.996543 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:00.997656 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:00.999577 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.000172 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.000303 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.002349 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:01.002357 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:01.002362 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:01.002400 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:01.005615 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:01.006426 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:01.006432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:01.006435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:01.006567 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:01.008752 (+ 2185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:01.008758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:01.008761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:01.009171 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:01.009325 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:01.010022 (+ 697us) server_negotiation.cc:299] Negotiation successful
0929 20:53:01.010266 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:53:01.010706 1841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:00.993092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39989 (local address 127.0.97.2:58029)
0929 20:53:00.994366 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:00.994392 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:00.994571 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:00.994982 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:00.994987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:00.995694 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:00.996323 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:00.996341 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:00.997782 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:00.997786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:00.999440 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:00.999453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.000369 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.000374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.000509 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.001125 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:01.001150 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:01.002145 (+ 995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:01.005744 (+ 3599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:01.005750 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:01.005754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:01.006295 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:01.006702 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:01.006707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:01.006711 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:01.008640 (+ 1929us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:01.009285 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:01.009290 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:01.009765 (+ 475us) client_negotiation.cc:769] Sending connection context
0929 20:53:01.009884 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 20:53:01.010271 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"thread_start_us":936,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:53:01.041199 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39989
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:01.355446 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:01.355731 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:01.355827 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:01.363013 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:01.363142 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:01.363233 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:53:01.363317 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:53:01.373261 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:01.374585 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:01.375922 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:01.388293 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:01.388315 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:01.388368 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:01.388958 1861 server_base.cc:1034] running on GCE node
I20260929 20:53:01.389570 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:01.390496 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:01.391693 1861 hybrid_clock.cc:648] HybridClock initialized: now 1790715181391629 us; error 43 us; skew 500 ppm
I20260929 20:53:01.391979 1861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:01 dist-test-slave-m9g1 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:01.394955 1861 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:53:01.396306 1861 webserver.cc:460] Webserver started at http://127.0.97.3:38127/ using document root <none> and password file <none>
I20260929 20:53:01.397006 1861 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:01.397089 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:01.397397 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:01.400241 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "3baf8a4084544614af931f0ce0884e8d"
format_stamp: "Formatted at 2026-09-29 20:53:01 on dist-test-slave-m9g1"
server_key: "007cddb6af9676c54cb80b5757e83a12"
server_key_iv: "37e077d162253da593e07112d578c907"
server_key_version: "encryptionkey@0"
I20260929 20:53:01.401016 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3baf8a4084544614af931f0ce0884e8d"
format_stamp: "Formatted at 2026-09-29 20:53:01 on dist-test-slave-m9g1"
server_key: "007cddb6af9676c54cb80b5757e83a12"
server_key_iv: "37e077d162253da593e07112d578c907"
server_key_version: "encryptionkey@0"
I20260929 20:53:01.405961 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 20:53:01.409301 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:01.410692 1861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:53:01.410840 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "3baf8a4084544614af931f0ce0884e8d"
format_stamp: "Formatted at 2026-09-29 20:53:01 on dist-test-slave-m9g1"
server_key: "007cddb6af9676c54cb80b5757e83a12"
server_key_iv: "37e077d162253da593e07112d578c907"
server_key_version: "encryptionkey@0"
I20260929 20:53:01.411031 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:01.440230 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:01.444631 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:01.444911 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:01.447810 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:01.447901 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:01.448022 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:01.448123 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 29 20:53:01 dist-test-slave-m9g1 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:01.471443 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.448299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60493 (local address 127.0.97.62:39989)
0929 20:53:01.448503 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:01.448509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:01.449986 (+ 1477us) server_negotiation.cc:407] Connection header received
0929 20:53:01.452099 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:01.452106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:01.452202 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:01.452323 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:01.454850 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.455823 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.458819 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.459399 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.459531 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.464400 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:01.464407 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:01.464412 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:01.464478 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:01.468066 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:01.468827 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:01.468833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:01.468836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:01.469019 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:01.469425 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:01.469430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:01.469434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:01.469800 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:01.469940 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:01.470704 (+ 764us) server_negotiation.cc:299] Negotiation successful
0929 20:53:01.470969 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 20:53:01.472349 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.448916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39989 (local address 127.0.97.3:60493)
0929 20:53:01.449782 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:01.449840 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:01.451756 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:01.452747 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:01.452756 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:01.453631 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:01.454655 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.454684 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.456002 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.456011 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.458594 (+ 2583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.458615 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.459855 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.459863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.460139 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.460985 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:01.461022 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:01.464145 (+ 3123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:01.468229 (+ 4084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:01.468240 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:01.468262 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:01.468707 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:01.469128 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:01.469132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:01.469134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:01.469328 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:01.469916 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:01.469922 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:01.470341 (+ 419us) client_negotiation.cc:769] Sending connection context
0929 20:53:01.470574 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 20:53:01.470976 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":358,"threads_started":1}
I20260929 20:53:01.491784 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36873
I20260929 20:53:01.491817 1995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36873 every 8 connection(s)
I20260929 20:53:01.493052 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:53:01.495162 1996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39989
I20260929 20:53:01.495388 1996 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:01.496119 1996 heartbeater.cc:511] Master 127.0.97.62:39989 requested a full tablet report, sending...
I20260929 20:53:01.497488 1511 ts_manager.cc:194] Registered new tserver with Master: 3baf8a4084544614af931f0ce0884e8d (127.0.97.3:36873)
I20260929 20:53:01.497741 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 1861
I20260929 20:53:01.497906 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:53:01.498432 388 external_mini_cluster.cc:1428] Setting key 2a56f79c85bc5cef6692217d7dc21038
I20260929 20:53:01.498523 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60493
I20260929 20:53:01.501286 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:53:01.502769 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.490021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47805 (local address 127.0.97.62:39989)
0929 20:53:01.490240 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:01.490245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:01.490304 (+ 59us) server_negotiation.cc:407] Connection header received
0929 20:53:01.490499 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:01.490502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:01.490564 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:01.490647 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:01.491821 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.492589 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.494056 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.494601 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.494713 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.496741 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:01.496748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:01.496753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:01.496792 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:01.500107 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:01.501138 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:01.501142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:01.501144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:01.501257 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:01.501610 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:01.501613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:01.501615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:01.501839 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:01.501936 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:01.502375 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 20:53:01.502529 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 20:53:01.502990 1991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.489503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39989 (local address 127.0.97.3:47805)
0929 20:53:01.490215 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:01.490236 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:01.490383 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:01.490700 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:01.490704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:01.491314 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:01.491666 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.491676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.492735 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.492741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.493857 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.493875 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.494956 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.494962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.495131 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.495750 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:01.495767 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:01.496542 (+ 775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:01.500258 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:01.500263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:01.500267 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:01.500756 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:01.501359 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:01.501364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:01.501367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:01.501507 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:01.501932 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:01.501937 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:01.502112 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 20:53:01.502287 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 20:53:01.502578 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":375,"threads_started":1}
Sep 29 20:53:01 dist-test-slave-m9g1 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715181, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260929 20:53:01.528676 1718 heartbeater.cc:503] Master 127.0.97.62:39989 was elected leader, sending a full tablet report...
Sep 29 20:53:01 dist-test-slave-m9g1 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715181, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:01.545308 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.532344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44392 (local address 127.0.97.62:39989)
0929 20:53:01.532488 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:01.532492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:01.532816 (+ 324us) server_negotiation.cc:407] Connection header received
0929 20:53:01.532968 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:01.532972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:01.533035 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:01.533149 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:01.534561 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.535324 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.536496 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.536912 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.537005 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.540592 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:01.540598 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:01.540602 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:01.540638 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:01.543063 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:01.543637 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:01.543641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:01.543643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:01.543769 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:01.544086 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:01.544090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:01.544092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:01.544385 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:01.544620 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:01.544783 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 20:53:01.544963 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 20:53:01.549269 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:53:01.556722 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:53:01.592772 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.568867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37124 (local address 127.0.97.2:38095)
0929 20:53:01.585238 (+ 16371us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:01.585247 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:01.585276 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:53:01.585363 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:01.585370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:01.585600 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:01.585818 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:01.586481 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.587729 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.590250 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.591640 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.591771 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.591973 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:01.592277 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 20:53:01.592453 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16020,"thread_start_us":16113,"threads_started":1}
I20260929 20:53:01.592854 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.568592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38095 (local address 127.0.0.1:37124)
0929 20:53:01.569258 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:01.569296 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:01.583893 (+ 14597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:01.585802 (+ 1909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:01.585807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:01.585835 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:01.586299 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.586312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.587852 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.587857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.590026 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.590040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.591827 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.591834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.592019 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.592066 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 20:53:01.592201 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 20:53:01.592403 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":5,"thread_start_us":255,"threads_started":1}
I20260929 20:53:01.593166 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.569092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41841 (local address 127.0.0.1:34816)
0929 20:53:01.584101 (+ 15009us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:01.584119 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:01.584234 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:01.585499 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:01.585509 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:01.585553 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:01.586023 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.586036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.588476 (+ 2440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.588486 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.590584 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.590601 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.592369 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.592375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.592559 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.592597 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 20:53:01.592660 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 20:53:01.592791 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13565,"mutex_wait_us":1,"thread_start_us":13692,"threads_started":1}
I20260929 20:53:01.593560 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.575878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36873 (local address 127.0.0.1:57682)
0929 20:53:01.582837 (+ 6959us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:01.582865 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:01.583893 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:01.587105 (+ 3212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:01.587110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:01.587135 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:01.587496 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.587506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.589279 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.589282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.591043 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.591054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.592607 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.592614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.592786 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.592822 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:53:01.592889 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 20:53:01.593015 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6817,"mutex_wait_us":26,"spinlock_wait_cycles":1921408,"thread_start_us":7088,"threads_started":1}
I20260929 20:53:01.594067 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.569533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34816 (local address 127.0.97.1:41841)
0929 20:53:01.584743 (+ 15210us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:01.584753 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:01.584784 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:53:01.584887 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:01.584895 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:01.585200 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:01.585399 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:01.586527 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.588338 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.590772 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.592154 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.593392 (+ 1238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.593589 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:01.593651 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 20:53:01.593791 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15030,"thread_start_us":14904,"threads_started":1}
I20260929 20:53:01.594555 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.570439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57682 (local address 127.0.97.3:36873)
0929 20:53:01.586539 (+ 16100us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:01.586546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:01.586576 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:53:01.586663 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:01.586670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:01.586909 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:01.587663 (+ 754us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:01.587955 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.589141 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.591223 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.592395 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.593757 (+ 1362us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.593979 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:01.594040 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 20:53:01.594211 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15936,"thread_start_us":15754,"threads_started":1}
I20260929 20:53:01.597118 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 517f354ae3f740968d62df5586d452a8 (DEFAULT_TABLE table=test-table [id=89f5eb23291a41dea734dff965f2f217]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:01.597576 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 517f354ae3f740968d62df5586d452a8 (DEFAULT_TABLE table=test-table [id=89f5eb23291a41dea734dff965f2f217]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:01.597599 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 517f354ae3f740968d62df5586d452a8 (DEFAULT_TABLE table=test-table [id=89f5eb23291a41dea734dff965f2f217]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:01.598628 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517f354ae3f740968d62df5586d452a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:01.598739 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517f354ae3f740968d62df5586d452a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:01.598875 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517f354ae3f740968d62df5586d452a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:01.609875 2015 tablet_bootstrap.cc:492] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d: Bootstrap starting.
I20260929 20:53:01.612727 2017 tablet_bootstrap.cc:492] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54: Bootstrap starting.
I20260929 20:53:01.613468 2016 tablet_bootstrap.cc:492] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003: Bootstrap starting.
I20260929 20:53:01.614388 2015 tablet_bootstrap.cc:654] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:01.615689 2015 log.cc:824] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:01.616902 2016 tablet_bootstrap.cc:654] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:01.618106 2016 log.cc:824] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:01.618330 2015 tablet_bootstrap.cc:492] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d: No bootstrap required, opened a new log
I20260929 20:53:01.618592 2015 ts_tablet_manager.cc:1397] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260929 20:53:01.620010 2017 tablet_bootstrap.cc:654] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:01.621161 2016 tablet_bootstrap.cc:492] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003: No bootstrap required, opened a new log
I20260929 20:53:01.621176 2017 log.cc:824] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:01.621420 2016 ts_tablet_manager.cc:1397] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 20:53:01.624270 2017 tablet_bootstrap.cc:492] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54: No bootstrap required, opened a new log
I20260929 20:53:01.624549 2017 ts_tablet_manager.cc:1397] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260929 20:53:01.627149 2015 raft_consensus.cc:348] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } } peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } }
I20260929 20:53:01.627486 2015 raft_consensus.cc:374] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:01.627602 2015 raft_consensus.cc:729] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3baf8a4084544614af931f0ce0884e8d, State: Initialized, Role: FOLLOWER
I20260929 20:53:01.628188 2015 consensus_queue.cc:260] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } } peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } }
I20260929 20:53:01.629253 1996 heartbeater.cc:503] Master 127.0.97.62:39989 was elected leader, sending a full tablet report...
I20260929 20:53:01.629927 2015 ts_tablet_manager.cc:1428] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260929 20:53:01.631332 2016 raft_consensus.cc:348] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } } peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } }
I20260929 20:53:01.631784 2016 raft_consensus.cc:374] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:01.631910 2016 raft_consensus.cc:729] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af7d71f8f7340c3b27d110f32891003, State: Initialized, Role: FOLLOWER
I20260929 20:53:01.632498 2016 consensus_queue.cc:260] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } } peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } }
I20260929 20:53:01.635681 2017 raft_consensus.cc:348] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } } peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } }
I20260929 20:53:01.636181 2017 raft_consensus.cc:374] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:01.636276 2017 raft_consensus.cc:729] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4664b98e374042c3970dee6322790e54, State: Initialized, Role: FOLLOWER
I20260929 20:53:01.636922 2017 consensus_queue.cc:260] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } } peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } }
I20260929 20:53:01.640125 2017 ts_tablet_manager.cc:1428] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54: Time spent starting tablet: real 0.015s user 0.002s sys 0.011s
I20260929 20:53:01.641762 1857 heartbeater.cc:503] Master 127.0.97.62:39989 was elected leader, sending a full tablet report...
I20260929 20:53:01.641644 2022 raft_consensus.cc:482] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:53:01.642489 2016 ts_tablet_manager.cc:1428] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003: Time spent starting tablet: real 0.021s user 0.015s sys 0.008s
I20260929 20:53:01.642771 2022 raft_consensus.cc:504] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } } peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } }
I20260929 20:53:01.644273 2022 leader_election.cc:288] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3baf8a4084544614af931f0ce0884e8d (127.0.97.3:36873), 4664b98e374042c3970dee6322790e54 (127.0.97.1:41841)
I20260929 20:53:01.652261 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.644696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37397 (local address 127.0.97.1:41841)
0929 20:53:01.644878 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:01.644883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:01.646411 (+ 1528us) server_negotiation.cc:407] Connection header received
0929 20:53:01.646643 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:01.646648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:01.646727 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:01.646779 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:01.647529 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.648225 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.650621 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.651400 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.651538 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.651633 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:01.651901 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 20:53:01.652024 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:53:01.652287 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.645050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54419 (local address 127.0.97.3:36873)
0929 20:53:01.645247 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:01.645253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:01.645868 (+ 615us) server_negotiation.cc:407] Connection header received
0929 20:53:01.646140 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:01.646146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:01.646253 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:01.646338 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:01.647072 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.648110 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.650632 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.651376 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.651488 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.651595 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:01.651918 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 20:53:01.652045 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:53:01.652419 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.644743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41841 (local address 127.0.97.2:37397)
0929 20:53:01.646327 (+ 1584us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:01.646343 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:01.646498 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:01.646921 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:01.646924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:01.646961 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:01.647347 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.647359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.648360 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.648365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.650479 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.650491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.651630 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.651636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.651754 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.651787 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:53:01.651862 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 20:53:01.651988 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1480,"mutex_wait_us":7,"spinlock_wait_cycles":1280,"thread_start_us":350,"threads_started":1}
I20260929 20:53:01.652639 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.644684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36873 (local address 127.0.97.2:54419)
0929 20:53:01.645736 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:01.645755 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:01.645988 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:01.646500 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:01.646505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:01.646529 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:01.646903 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.646914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.648254 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.648259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.650473 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.650485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.651617 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.651623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.651769 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.651801 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:53:01.651879 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 20:53:01.652001 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"mutex_wait_us":52,"thread_start_us":572,"threads_started":1}
I20260929 20:53:01.653404 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517f354ae3f740968d62df5586d452a8" candidate_uuid: "3af7d71f8f7340c3b27d110f32891003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4664b98e374042c3970dee6322790e54" is_pre_election: true
I20260929 20:53:01.653539 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517f354ae3f740968d62df5586d452a8" candidate_uuid: "3af7d71f8f7340c3b27d110f32891003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3baf8a4084544614af931f0ce0884e8d" is_pre_election: true
I20260929 20:53:01.653990 1946 raft_consensus.cc:2454] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3af7d71f8f7340c3b27d110f32891003 in term 0.
I20260929 20:53:01.653977 1668 raft_consensus.cc:2454] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3af7d71f8f7340c3b27d110f32891003 in term 0.
I20260929 20:53:01.654644 1739 leader_election.cc:302] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af7d71f8f7340c3b27d110f32891003, 4664b98e374042c3970dee6322790e54; no voters:
I20260929 20:53:01.654989 2022 raft_consensus.cc:2781] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:53:01.655131 2022 raft_consensus.cc:482] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:53:01.655236 2022 raft_consensus.cc:3037] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:01.657505 2022 raft_consensus.cc:504] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } } peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } }
I20260929 20:53:01.658094 2022 leader_election.cc:288] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [CANDIDATE]: Term 1 election: Requested vote from peers 3baf8a4084544614af931f0ce0884e8d (127.0.97.3:36873), 4664b98e374042c3970dee6322790e54 (127.0.97.1:41841)
I20260929 20:53:01.658531 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517f354ae3f740968d62df5586d452a8" candidate_uuid: "3af7d71f8f7340c3b27d110f32891003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3baf8a4084544614af931f0ce0884e8d"
I20260929 20:53:01.658625 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517f354ae3f740968d62df5586d452a8" candidate_uuid: "3af7d71f8f7340c3b27d110f32891003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4664b98e374042c3970dee6322790e54"
I20260929 20:53:01.658710 1946 raft_consensus.cc:3037] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:01.658828 1668 raft_consensus.cc:3037] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:01.661124 1946 raft_consensus.cc:2454] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3af7d71f8f7340c3b27d110f32891003 in term 1.
I20260929 20:53:01.661511 1738 leader_election.cc:302] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af7d71f8f7340c3b27d110f32891003, 3baf8a4084544614af931f0ce0884e8d; no voters:
I20260929 20:53:01.661566 1668 raft_consensus.cc:2454] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3af7d71f8f7340c3b27d110f32891003 in term 1.
I20260929 20:53:01.661818 2022 raft_consensus.cc:2781] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:01.661999 2022 raft_consensus.cc:686] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [term 1 LEADER]: Becoming Leader. State: Replica: 3af7d71f8f7340c3b27d110f32891003, State: Running, Role: LEADER
I20260929 20:53:01.662552 2022 consensus_queue.cc:237] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } } peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } }
I20260929 20:53:01.666745 1510 catalog_manager.cc:5495] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3af7d71f8f7340c3b27d110f32891003 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3af7d71f8f7340c3b27d110f32891003" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } health_report { overall_health: UNKNOWN } } }
I20260929 20:53:01.699368 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.694366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37128 (local address 127.0.97.2:38095)
0929 20:53:01.694610 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:01.694616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:01.694656 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:53:01.694936 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:01.694939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:01.695025 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:01.695141 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:01.695428 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.696150 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.697416 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.697812 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.697903 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.698340 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:01.698866 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:01.698969 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:01.699035 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 20:53:01.699103 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:53:01.707551 1946 raft_consensus.cc:1261] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d [term 1 FOLLOWER]: Refusing update from remote peer 3af7d71f8f7340c3b27d110f32891003: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:01.707551 1668 raft_consensus.cc:1261] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54 [term 1 FOLLOWER]: Refusing update from remote peer 3af7d71f8f7340c3b27d110f32891003: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:01.708458 2023 consensus_queue.cc:1035] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:01.708791 2022 consensus_queue.cc:1035] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:01.724805 2031 mvcc.cc:205] Tried to move back new op lower bound from 7334769384265551872 to 7334769384094072832. Current Snapshot: MvccSnapshot[applied={T|T < 7334769384265551872}]
I20260929 20:53:01.737421 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:01.737684 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:01.741197 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d: Sending DeleteTablet for 3 replicas of tablet 517f354ae3f740968d62df5586d452a8
I20260929 20:53:01.742393 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 517f354ae3f740968d62df5586d452a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37124
I20260929 20:53:01.742445 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 517f354ae3f740968d62df5586d452a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57682
I20260929 20:53:01.742574 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 517f354ae3f740968d62df5586d452a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34816
I20260929 20:53:01.745066 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:53:01.746369 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:53:01.755402 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb04c9a08a847f5bcf4095eaaaff134 (DEFAULT_TABLE table=test-table [id=0a4f852708314291a9dc5f4ef830e4ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:01.755808 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb04c9a08a847f5bcf4095eaaaff134 (DEFAULT_TABLE table=test-table [id=0a4f852708314291a9dc5f4ef830e4ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:01.755990 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb04c9a08a847f5bcf4095eaaaff134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:01.755996 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb04c9a08a847f5bcf4095eaaaff134 (DEFAULT_TABLE table=test-table [id=0a4f852708314291a9dc5f4ef830e4ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:01.756311 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb04c9a08a847f5bcf4095eaaaff134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:01.756462 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb04c9a08a847f5bcf4095eaaaff134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:01.761567 2041 tablet_replica.cc:330] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54: stopping tablet replica
I20260929 20:53:01.762024 2041 raft_consensus.cc:2229] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:01.762957 2041 raft_consensus.cc:2258] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:01.762997 2015 tablet_bootstrap.cc:492] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d: Bootstrap starting.
I20260929 20:53:01.763499 2017 tablet_bootstrap.cc:492] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54: Bootstrap starting.
I20260929 20:53:01.764138 2016 tablet_bootstrap.cc:492] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003: Bootstrap starting.
I20260929 20:53:01.764768 2015 tablet_bootstrap.cc:654] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:01.765408 2017 tablet_bootstrap.cc:654] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:01.765631 2016 tablet_bootstrap.cc:654] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:01.769383 2039 tablet_replica.cc:330] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d: stopping tablet replica
I20260929 20:53:01.770407 2039 raft_consensus.cc:2229] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:01.772956 2039 raft_consensus.cc:2258] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:01.773926 2040 tablet_replica.cc:330] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003: stopping tablet replica
I20260929 20:53:01.774441 2040 raft_consensus.cc:2229] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:53:01.777674 2041 ts_tablet_manager.cc:1905] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:01.777732 2040 raft_consensus.cc:2258] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:01.780306 2039 ts_tablet_manager.cc:1905] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:01.780406 2017 tablet_bootstrap.cc:492] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54: No bootstrap required, opened a new log
I20260929 20:53:01.780710 2017 ts_tablet_manager.cc:1397] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20260929 20:53:01.781505 2017 raft_consensus.cc:348] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } }
I20260929 20:53:01.781729 2017 raft_consensus.cc:374] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:01.781826 2017 raft_consensus.cc:729] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4664b98e374042c3970dee6322790e54, State: Initialized, Role: FOLLOWER
I20260929 20:53:01.781876 2016 tablet_bootstrap.cc:492] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003: No bootstrap required, opened a new log
I20260929 20:53:01.782022 2016 ts_tablet_manager.cc:1397] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.007s
I20260929 20:53:01.782091 2017 consensus_queue.cc:260] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } }
I20260929 20:53:01.782625 2017 ts_tablet_manager.cc:1428] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:53:01.786985 2040 ts_tablet_manager.cc:1905] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:01.787943 2016 raft_consensus.cc:348] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } }
I20260929 20:53:01.788178 2016 raft_consensus.cc:374] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:01.788275 2016 raft_consensus.cc:729] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af7d71f8f7340c3b27d110f32891003, State: Initialized, Role: FOLLOWER
I20260929 20:53:01.788582 2016 consensus_queue.cc:260] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } }
I20260929 20:53:01.789155 2016 ts_tablet_manager.cc:1428] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260929 20:53:01.793967 2015 tablet_bootstrap.cc:492] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d: No bootstrap required, opened a new log
I20260929 20:53:01.794222 2015 ts_tablet_manager.cc:1397] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.003s
I20260929 20:53:01.795243 2041 ts_tablet_manager.cc:1918] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:01.795243 2015 raft_consensus.cc:348] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } }
I20260929 20:53:01.795464 2041 log.cc:1196] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/517f354ae3f740968d62df5586d452a8
I20260929 20:53:01.795640 2015 raft_consensus.cc:374] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:01.795840 2015 raft_consensus.cc:729] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3baf8a4084544614af931f0ce0884e8d, State: Initialized, Role: FOLLOWER
I20260929 20:53:01.796013 2041 ts_tablet_manager.cc:1939] T 517f354ae3f740968d62df5586d452a8 P 4664b98e374042c3970dee6322790e54: Deleting consensus metadata
I20260929 20:53:01.796232 2015 consensus_queue.cc:260] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } }
I20260929 20:53:01.796518 2040 ts_tablet_manager.cc:1918] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:01.796716 2040 log.cc:1196] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/517f354ae3f740968d62df5586d452a8
I20260929 20:53:01.797223 2040 ts_tablet_manager.cc:1939] T 517f354ae3f740968d62df5586d452a8 P 3af7d71f8f7340c3b27d110f32891003: Deleting consensus metadata
I20260929 20:53:01.798048 1496 catalog_manager.cc:4841] TS 4664b98e374042c3970dee6322790e54 (127.0.97.1:41841): tablet 517f354ae3f740968d62df5586d452a8 (table test-table [id=89f5eb23291a41dea734dff965f2f217]) successfully deleted
I20260929 20:53:01.798671 2039 ts_tablet_manager.cc:1918] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:01.798826 2039 log.cc:1196] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/517f354ae3f740968d62df5586d452a8
I20260929 20:53:01.798812 1495 catalog_manager.cc:4841] TS 3af7d71f8f7340c3b27d110f32891003 (127.0.97.2:38095): tablet 517f354ae3f740968d62df5586d452a8 (table test-table [id=89f5eb23291a41dea734dff965f2f217]) successfully deleted
I20260929 20:53:01.799362 2039 ts_tablet_manager.cc:1939] T 517f354ae3f740968d62df5586d452a8 P 3baf8a4084544614af931f0ce0884e8d: Deleting consensus metadata
I20260929 20:53:01.799415 2015 ts_tablet_manager.cc:1428] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260929 20:53:01.801098 1495 catalog_manager.cc:4841] TS 3baf8a4084544614af931f0ce0884e8d (127.0.97.3:36873): tablet 517f354ae3f740968d62df5586d452a8 (table test-table [id=89f5eb23291a41dea734dff965f2f217]) successfully deleted
I20260929 20:53:01.839778 2024 raft_consensus.cc:482] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:53:01.840011 2024 raft_consensus.cc:504] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } }
I20260929 20:53:01.841900 2024 leader_election.cc:288] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3af7d71f8f7340c3b27d110f32891003 (127.0.97.2:38095), 3baf8a4084544614af931f0ce0884e8d (127.0.97.3:36873)
I20260929 20:53:01.842020 2023 raft_consensus.cc:482] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:53:01.842267 2023 raft_consensus.cc:504] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } }
I20260929 20:53:01.842959 2023 leader_election.cc:288] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4664b98e374042c3970dee6322790e54 (127.0.97.1:41841), 3baf8a4084544614af931f0ce0884e8d (127.0.97.3:36873)
I20260929 20:53:01.843896 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb04c9a08a847f5bcf4095eaaaff134" candidate_uuid: "3af7d71f8f7340c3b27d110f32891003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3baf8a4084544614af931f0ce0884e8d" is_pre_election: true
I20260929 20:53:01.844174 1946 raft_consensus.cc:2454] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3af7d71f8f7340c3b27d110f32891003 in term 0.
I20260929 20:53:01.844792 1738 leader_election.cc:302] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af7d71f8f7340c3b27d110f32891003, 3baf8a4084544614af931f0ce0884e8d; no voters:
I20260929 20:53:01.845137 2023 raft_consensus.cc:2781] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:53:01.845237 2023 raft_consensus.cc:482] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:53:01.845803 2023 raft_consensus.cc:3037] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:01.845943 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb04c9a08a847f5bcf4095eaaaff134" candidate_uuid: "3af7d71f8f7340c3b27d110f32891003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4664b98e374042c3970dee6322790e54" is_pre_election: true
I20260929 20:53:01.846163 1668 raft_consensus.cc:2454] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3af7d71f8f7340c3b27d110f32891003 in term 0.
I20260929 20:53:01.848973 2023 raft_consensus.cc:504] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } }
I20260929 20:53:01.850055 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb04c9a08a847f5bcf4095eaaaff134" candidate_uuid: "3af7d71f8f7340c3b27d110f32891003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4664b98e374042c3970dee6322790e54"
I20260929 20:53:01.850171 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb04c9a08a847f5bcf4095eaaaff134" candidate_uuid: "3af7d71f8f7340c3b27d110f32891003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3baf8a4084544614af931f0ce0884e8d"
I20260929 20:53:01.850281 1668 raft_consensus.cc:3037] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:01.850402 1946 raft_consensus.cc:3037] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:01.851244 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.842419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53477 (local address 127.0.97.2:38095)
0929 20:53:01.842599 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:01.842605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:01.843361 (+ 756us) server_negotiation.cc:407] Connection header received
0929 20:53:01.843629 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:01.843635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:01.843716 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:01.843781 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:01.844679 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.846316 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.848738 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.849956 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.850075 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.850169 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:01.850788 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 20:53:01.850936 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:53:01.851953 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.842166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38095 (local address 127.0.97.1:53477)
0929 20:53:01.843254 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:01.843277 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:01.843469 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:01.843976 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:01.843981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:01.844008 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:01.844505 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.844517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.846468 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.846473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.848551 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.848565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.850400 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.850406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.850600 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.850635 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 20:53:01.851470 (+ 835us) client_negotiation.cc:240] Negotiation successful
0929 20:53:01.851614 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"thread_start_us":702,"threads_started":1}
I20260929 20:53:01.852330 2023 leader_election.cc:288] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [CANDIDATE]: Term 1 election: Requested vote from peers 4664b98e374042c3970dee6322790e54 (127.0.97.1:41841), 3baf8a4084544614af931f0ce0884e8d (127.0.97.3:36873)
I20260929 20:53:01.853236 1946 raft_consensus.cc:2454] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3af7d71f8f7340c3b27d110f32891003 in term 1.
I20260929 20:53:01.853382 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb04c9a08a847f5bcf4095eaaaff134" candidate_uuid: "4664b98e374042c3970dee6322790e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af7d71f8f7340c3b27d110f32891003" is_pre_election: true
I20260929 20:53:01.853783 1738 leader_election.cc:302] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af7d71f8f7340c3b27d110f32891003, 3baf8a4084544614af931f0ce0884e8d; no voters:
I20260929 20:53:01.854107 1807 raft_consensus.cc:2379] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4664b98e374042c3970dee6322790e54 in current term 1: Already voted for candidate 3af7d71f8f7340c3b27d110f32891003 in this term.
I20260929 20:53:01.854471 1668 raft_consensus.cc:2454] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3af7d71f8f7340c3b27d110f32891003 in term 1.
I20260929 20:53:01.854776 2023 raft_consensus.cc:2781] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:01.854951 2023 raft_consensus.cc:686] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [term 1 LEADER]: Becoming Leader. State: Replica: 3af7d71f8f7340c3b27d110f32891003, State: Running, Role: LEADER
I20260929 20:53:01.855406 2023 consensus_queue.cc:237] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } }
I20260929 20:53:01.857963 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.843322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36873 (local address 127.0.97.1:56969)
0929 20:53:01.845727 (+ 2405us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:01.845744 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:01.845899 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:01.846425 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:01.846429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:01.846453 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:01.846865 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.846875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.849304 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.849309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.855607 (+ 6298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.855619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.857166 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.857172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.857338 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.857371 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:53:01.857463 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 20:53:01.857606 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2295,"mutex_wait_us":3,"thread_start_us":2118,"threads_started":1}
I20260929 20:53:01.858290 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.843653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56969 (local address 127.0.97.3:36873)
0929 20:53:01.844486 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:01.844492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:01.845818 (+ 1326us) server_negotiation.cc:407] Connection header received
0929 20:53:01.846063 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:01.846068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:01.846151 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:01.846238 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:01.847027 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.848055 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.855827 (+ 7772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.856808 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.856931 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.857026 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:01.857734 (+ 708us) server_negotiation.cc:299] Negotiation successful
0929 20:53:01.857865 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720}
I20260929 20:53:01.858151 1510 catalog_manager.cc:5495] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3af7d71f8f7340c3b27d110f32891003 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3af7d71f8f7340c3b27d110f32891003" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } health_report { overall_health: UNKNOWN } } }
I20260929 20:53:01.858807 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb04c9a08a847f5bcf4095eaaaff134" candidate_uuid: "4664b98e374042c3970dee6322790e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3baf8a4084544614af931f0ce0884e8d" is_pre_election: true
I20260929 20:53:01.859056 1946 raft_consensus.cc:2379] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4664b98e374042c3970dee6322790e54 in current term 1: Already voted for candidate 3af7d71f8f7340c3b27d110f32891003 in this term.
I20260929 20:53:01.859453 1598 leader_election.cc:302] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4664b98e374042c3970dee6322790e54; no voters: 3af7d71f8f7340c3b27d110f32891003, 3baf8a4084544614af931f0ce0884e8d
I20260929 20:53:01.859875 2024 raft_consensus.cc:2726] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 29 20:53:01 dist-test-slave-m9g1 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:01.910614 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.896686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44396 (local address 127.0.97.62:39989)
0929 20:53:01.896887 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:01.896894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:01.897434 (+ 540us) server_negotiation.cc:407] Connection header received
0929 20:53:01.897584 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:01.897589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:01.897667 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:01.897779 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:01.899066 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.900101 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.901480 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.902151 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.902296 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.905588 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:01.905593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:01.905597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:01.905625 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:01.907981 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:01.908825 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:01.908829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:01.908832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:01.908937 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:01.909299 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:01.909302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:01.909304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:01.909550 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:01.909651 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:01.910126 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 20:53:01.910358 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":78}
I20260929 20:53:01.911088 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:01.896690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39989 (local address 127.0.0.1:44396)
0929 20:53:01.897323 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:01.897343 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:01.897477 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:01.897831 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:01.897835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:01.898644 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:01.898917 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.898925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.900249 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.900253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.901336 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:01.901349 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:01.902334 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:01.902338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:01.902452 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:01.903056 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:01.903078 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:01.905410 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:01.908120 (+ 2710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:01.908134 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:01.908142 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:01.908720 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:01.909031 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:01.909036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:01.909040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:01.909200 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:01.909663 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:01.909672 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:01.909869 (+ 197us) client_negotiation.cc:769] Sending connection context
0929 20:53:01.910040 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 20:53:01.910513 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":125,"thread_start_us":373,"threads_started":1}
I20260929 20:53:01.916631 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44396:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 20:53:01.917876 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:53:01.924404 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 8a78f6c5a852443bbd2fa0d374a32895 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5f8be708b80e43128f4231b80c4bedb2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:53:01.924384 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 8a78f6c5a852443bbd2fa0d374a32895 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5f8be708b80e43128f4231b80c4bedb2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:53:01.924952 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a78f6c5a852443bbd2fa0d374a32895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:01.924975 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a78f6c5a852443bbd2fa0d374a32895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:01.927301 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 8a78f6c5a852443bbd2fa0d374a32895 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5f8be708b80e43128f4231b80c4bedb2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:53:01.927806 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a78f6c5a852443bbd2fa0d374a32895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:01.932166 2017 tablet_bootstrap.cc:492] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54: Bootstrap starting.
I20260929 20:53:01.933844 2017 tablet_bootstrap.cc:654] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:01.936816 2015 tablet_bootstrap.cc:492] T 8a78f6c5a852443bbd2fa0d374a32895 P 3baf8a4084544614af931f0ce0884e8d: Bootstrap starting.
I20260929 20:53:01.939204 2015 tablet_bootstrap.cc:654] T 8a78f6c5a852443bbd2fa0d374a32895 P 3baf8a4084544614af931f0ce0884e8d: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:01.942117 2016 tablet_bootstrap.cc:492] T 8a78f6c5a852443bbd2fa0d374a32895 P 3af7d71f8f7340c3b27d110f32891003: Bootstrap starting.
I20260929 20:53:01.943888 2016 tablet_bootstrap.cc:654] T 8a78f6c5a852443bbd2fa0d374a32895 P 3af7d71f8f7340c3b27d110f32891003: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:01.951509 2015 tablet_bootstrap.cc:492] T 8a78f6c5a852443bbd2fa0d374a32895 P 3baf8a4084544614af931f0ce0884e8d: No bootstrap required, opened a new log
I20260929 20:53:01.951727 2015 ts_tablet_manager.cc:1397] T 8a78f6c5a852443bbd2fa0d374a32895 P 3baf8a4084544614af931f0ce0884e8d: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.005s
I20260929 20:53:01.952600 2015 raft_consensus.cc:348] T 8a78f6c5a852443bbd2fa0d374a32895 P 3baf8a4084544614af931f0ce0884e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } }
I20260929 20:53:01.952879 2015 raft_consensus.cc:374] T 8a78f6c5a852443bbd2fa0d374a32895 P 3baf8a4084544614af931f0ce0884e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:01.952953 2015 raft_consensus.cc:729] T 8a78f6c5a852443bbd2fa0d374a32895 P 3baf8a4084544614af931f0ce0884e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3baf8a4084544614af931f0ce0884e8d, State: Initialized, Role: FOLLOWER
I20260929 20:53:01.953233 2015 consensus_queue.cc:260] T 8a78f6c5a852443bbd2fa0d374a32895 P 3baf8a4084544614af931f0ce0884e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } }
I20260929 20:53:01.953794 2015 ts_tablet_manager.cc:1428] T 8a78f6c5a852443bbd2fa0d374a32895 P 3baf8a4084544614af931f0ce0884e8d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 20:53:01.954493 2021 tablet_replica.cc:439] T 8a78f6c5a852443bbd2fa0d374a32895 P 3baf8a4084544614af931f0ce0884e8d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } } }
I20260929 20:53:01.954900 2021 tablet_replica.cc:442] T 8a78f6c5a852443bbd2fa0d374a32895 P 3baf8a4084544614af931f0ce0884e8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:01.955003 2016 tablet_bootstrap.cc:492] T 8a78f6c5a852443bbd2fa0d374a32895 P 3af7d71f8f7340c3b27d110f32891003: No bootstrap required, opened a new log
I20260929 20:53:01.955160 2017 tablet_bootstrap.cc:492] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54: No bootstrap required, opened a new log
I20260929 20:53:01.955193 2016 ts_tablet_manager.cc:1397] T 8a78f6c5a852443bbd2fa0d374a32895 P 3af7d71f8f7340c3b27d110f32891003: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260929 20:53:01.955345 2017 ts_tablet_manager.cc:1397] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.012s
I20260929 20:53:01.955925 2016 raft_consensus.cc:348] T 8a78f6c5a852443bbd2fa0d374a32895 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } }
I20260929 20:53:01.955998 2017 raft_consensus.cc:348] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } }
I20260929 20:53:01.956177 2016 raft_consensus.cc:374] T 8a78f6c5a852443bbd2fa0d374a32895 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:01.956197 2017 raft_consensus.cc:374] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:01.956254 2016 raft_consensus.cc:729] T 8a78f6c5a852443bbd2fa0d374a32895 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af7d71f8f7340c3b27d110f32891003, State: Initialized, Role: FOLLOWER
I20260929 20:53:01.956260 2017 raft_consensus.cc:729] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4664b98e374042c3970dee6322790e54, State: Initialized, Role: FOLLOWER
I20260929 20:53:01.956502 2016 consensus_queue.cc:260] T 8a78f6c5a852443bbd2fa0d374a32895 P 3af7d71f8f7340c3b27d110f32891003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } }
I20260929 20:53:01.956555 2017 consensus_queue.cc:260] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } }
I20260929 20:53:01.957093 2016 ts_tablet_manager.cc:1428] T 8a78f6c5a852443bbd2fa0d374a32895 P 3af7d71f8f7340c3b27d110f32891003: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 20:53:01.956945 2024 tablet_replica.cc:439] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } } }
I20260929 20:53:01.957199 2024 tablet_replica.cc:442] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:01.957118 2023 tablet_replica.cc:439] T 8a78f6c5a852443bbd2fa0d374a32895 P 3af7d71f8f7340c3b27d110f32891003: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } } }
I20260929 20:53:01.957381 2023 tablet_replica.cc:442] T 8a78f6c5a852443bbd2fa0d374a32895 P 3af7d71f8f7340c3b27d110f32891003: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:01.958516 2017 ts_tablet_manager.cc:1428] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260929 20:53:01.995954 1997 tablet.cc:2369] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:53:02.003609 1858 tablet.cc:2369] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:53:02.004204 2024 raft_consensus.cc:482] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:53:02.004376 2024 raft_consensus.cc:504] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } }
I20260929 20:53:02.005060 2024 leader_election.cc:288] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3af7d71f8f7340c3b27d110f32891003 (127.0.97.2:38095), 3baf8a4084544614af931f0ce0884e8d (127.0.97.3:36873)
I20260929 20:53:02.005790 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a78f6c5a852443bbd2fa0d374a32895" candidate_uuid: "4664b98e374042c3970dee6322790e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3baf8a4084544614af931f0ce0884e8d" is_pre_election: true
I20260929 20:53:02.006031 1946 raft_consensus.cc:2454] T 8a78f6c5a852443bbd2fa0d374a32895 P 3baf8a4084544614af931f0ce0884e8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4664b98e374042c3970dee6322790e54 in term 0.
I20260929 20:53:02.005993 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a78f6c5a852443bbd2fa0d374a32895" candidate_uuid: "4664b98e374042c3970dee6322790e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af7d71f8f7340c3b27d110f32891003" is_pre_election: true
I20260929 20:53:02.006500 1598 leader_election.cc:302] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3baf8a4084544614af931f0ce0884e8d, 4664b98e374042c3970dee6322790e54; no voters:
I20260929 20:53:02.006515 1807 raft_consensus.cc:2454] T 8a78f6c5a852443bbd2fa0d374a32895 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4664b98e374042c3970dee6322790e54 in term 0.
I20260929 20:53:02.006937 2024 raft_consensus.cc:2781] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:53:02.007053 2024 raft_consensus.cc:482] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:53:02.007125 2024 raft_consensus.cc:3037] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:02.010221 2024 raft_consensus.cc:504] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } }
I20260929 20:53:02.010869 2024 leader_election.cc:288] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 [CANDIDATE]: Term 1 election: Requested vote from peers 3af7d71f8f7340c3b27d110f32891003 (127.0.97.2:38095), 3baf8a4084544614af931f0ce0884e8d (127.0.97.3:36873)
I20260929 20:53:02.011511 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a78f6c5a852443bbd2fa0d374a32895" candidate_uuid: "4664b98e374042c3970dee6322790e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3baf8a4084544614af931f0ce0884e8d"
I20260929 20:53:02.011569 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a78f6c5a852443bbd2fa0d374a32895" candidate_uuid: "4664b98e374042c3970dee6322790e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af7d71f8f7340c3b27d110f32891003"
I20260929 20:53:02.011735 1946 raft_consensus.cc:3037] T 8a78f6c5a852443bbd2fa0d374a32895 P 3baf8a4084544614af931f0ce0884e8d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:02.011793 1807 raft_consensus.cc:3037] T 8a78f6c5a852443bbd2fa0d374a32895 P 3af7d71f8f7340c3b27d110f32891003 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:02.014708 1946 raft_consensus.cc:2454] T 8a78f6c5a852443bbd2fa0d374a32895 P 3baf8a4084544614af931f0ce0884e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4664b98e374042c3970dee6322790e54 in term 1.
I20260929 20:53:02.015298 1598 leader_election.cc:302] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3baf8a4084544614af931f0ce0884e8d, 4664b98e374042c3970dee6322790e54; no voters:
I20260929 20:53:02.015679 2024 raft_consensus.cc:2781] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:02.015887 1807 raft_consensus.cc:2454] T 8a78f6c5a852443bbd2fa0d374a32895 P 3af7d71f8f7340c3b27d110f32891003 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4664b98e374042c3970dee6322790e54 in term 1.
I20260929 20:53:02.016693 2024 raft_consensus.cc:686] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 [term 1 LEADER]: Becoming Leader. State: Replica: 4664b98e374042c3970dee6322790e54, State: Running, Role: LEADER
I20260929 20:53:02.017172 2024 consensus_queue.cc:237] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } }
I20260929 20:53:02.019344 2057 tablet_replica.cc:439] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54: TxnStatusTablet state changed. Reason: New leader 4664b98e374042c3970dee6322790e54. Latest consensus state: current_term: 1 leader_uuid: "4664b98e374042c3970dee6322790e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } } }
I20260929 20:53:02.019704 2057 tablet_replica.cc:442] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:53:02.020773 2059 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 20:53:02.021050 2059 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260929 20:53:02.022061 1719 tablet.cc:2369] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:53:02.022344 1508 catalog_manager.cc:5495] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4664b98e374042c3970dee6322790e54 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "4664b98e374042c3970dee6322790e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } health_report { overall_health: UNKNOWN } } }
I20260929 20:53:02.054786 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:02.049233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41841 (local address 127.0.0.1:34830)
0929 20:53:02.049556 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:02.049570 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:02.049703 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:02.050126 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:02.050130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:02.050141 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:53:02.050347 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:02.050356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.051381 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.051387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:02.052366 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:02.052378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.053157 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.053162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:02.053356 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:02.053411 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:53:02.054244 (+ 833us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:53:02.054263 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 20:53:02.054345 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 20:53:02.054493 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260929 20:53:02.054788 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:02.049234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34830 (local address 127.0.97.1:41841)
0929 20:53:02.049402 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:02.049408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:02.049640 (+ 232us) server_negotiation.cc:407] Connection header received
0929 20:53:02.049898 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:02.049902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:02.049975 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:02.050091 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:02.050479 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.051244 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.052495 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.053026 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.053134 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:02.053584 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:02.054103 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:02.054261 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:02.054357 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 20:53:02.054443 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:53:02.059087 1946 raft_consensus.cc:1261] T 8a78f6c5a852443bbd2fa0d374a32895 P 3baf8a4084544614af931f0ce0884e8d [term 1 FOLLOWER]: Refusing update from remote peer 4664b98e374042c3970dee6322790e54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:02.059314 1807 raft_consensus.cc:1261] T 8a78f6c5a852443bbd2fa0d374a32895 P 3af7d71f8f7340c3b27d110f32891003 [term 1 FOLLOWER]: Refusing update from remote peer 4664b98e374042c3970dee6322790e54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:02.059687 2057 consensus_queue.cc:1035] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:02.059979 2024 consensus_queue.cc:1035] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:02.073447 2021 tablet_replica.cc:439] T 8a78f6c5a852443bbd2fa0d374a32895 P 3baf8a4084544614af931f0ce0884e8d: TxnStatusTablet state changed. Reason: New leader 4664b98e374042c3970dee6322790e54. Latest consensus state: current_term: 1 leader_uuid: "4664b98e374042c3970dee6322790e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } } }
I20260929 20:53:02.073760 2021 tablet_replica.cc:442] T 8a78f6c5a852443bbd2fa0d374a32895 P 3baf8a4084544614af931f0ce0884e8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:02.074335 2023 tablet_replica.cc:439] T 8a78f6c5a852443bbd2fa0d374a32895 P 3af7d71f8f7340c3b27d110f32891003: TxnStatusTablet state changed. Reason: New leader 4664b98e374042c3970dee6322790e54. Latest consensus state: current_term: 1 leader_uuid: "4664b98e374042c3970dee6322790e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } } }
I20260929 20:53:02.074625 2023 tablet_replica.cc:442] T 8a78f6c5a852443bbd2fa0d374a32895 P 3af7d71f8f7340c3b27d110f32891003: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:02.077348 2024 tablet_replica.cc:439] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4664b98e374042c3970dee6322790e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } } }
I20260929 20:53:02.077641 2024 tablet_replica.cc:442] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:53:02.079624 2021 tablet_replica.cc:439] T 8a78f6c5a852443bbd2fa0d374a32895 P 3baf8a4084544614af931f0ce0884e8d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4664b98e374042c3970dee6322790e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } } }
I20260929 20:53:02.079865 2021 tablet_replica.cc:442] T 8a78f6c5a852443bbd2fa0d374a32895 P 3baf8a4084544614af931f0ce0884e8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:02.099320 2023 tablet_replica.cc:439] T 8a78f6c5a852443bbd2fa0d374a32895 P 3af7d71f8f7340c3b27d110f32891003: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4664b98e374042c3970dee6322790e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } } }
I20260929 20:53:02.099638 2023 tablet_replica.cc:442] T 8a78f6c5a852443bbd2fa0d374a32895 P 3af7d71f8f7340c3b27d110f32891003: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:02.103703 2057 tablet_replica.cc:439] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4664b98e374042c3970dee6322790e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 } } peers { permanent_uuid: "3af7d71f8f7340c3b27d110f32891003" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38095 } } peers { permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 } } }
I20260929 20:53:02.104017 2057 tablet_replica.cc:442] T 8a78f6c5a852443bbd2fa0d374a32895 P 4664b98e374042c3970dee6322790e54: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:53:02.116333 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:02.109990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35363 (local address 127.0.97.1:41841)
0929 20:53:02.110252 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:02.110260 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:02.111263 (+ 1003us) server_negotiation.cc:407] Connection header received
0929 20:53:02.111455 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:02.111460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:02.111543 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:02.111615 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:02.112148 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.112912 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.114501 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.115083 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.115214 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:02.115627 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:02.115749 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:02.115838 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:02.115943 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 20:53:02.116020 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:53:02.116668 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:02.110500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41841 (local address 127.0.97.2:35363)
0929 20:53:02.111150 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:02.111167 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:02.111319 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:02.111643 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:02.111648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:02.111659 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:53:02.112012 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:02.112021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.113030 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.113035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:02.114378 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:02.114390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.115251 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.115258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:02.115410 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:02.115464 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:53:02.115836 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:53:02.115847 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:53:02.115923 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 20:53:02.116054 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":345,"threads_started":1}
I20260929 20:53:02.127554 1946 raft_consensus.cc:1261] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d [term 1 FOLLOWER]: Refusing update from remote peer 3af7d71f8f7340c3b27d110f32891003: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:02.127558 1668 raft_consensus.cc:1261] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54 [term 1 FOLLOWER]: Refusing update from remote peer 3af7d71f8f7340c3b27d110f32891003: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:02.128147 2034 consensus_queue.cc:1035] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3baf8a4084544614af931f0ce0884e8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:02.128953 2023 consensus_queue.cc:1035] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4664b98e374042c3970dee6322790e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:02.142746 2029 mvcc.cc:205] Tried to move back new op lower bound from 7334769385989758976 to 7334769384883355648. Current Snapshot: MvccSnapshot[applied={T|T < 7334769385989758976}]
I20260929 20:53:02.201201 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:02.182461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53035 (local address 127.0.97.2:38095)
0929 20:53:02.182647 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:02.182654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:02.195924 (+ 13270us) server_negotiation.cc:407] Connection header received
0929 20:53:02.196234 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:02.196240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:02.196360 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:02.196423 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:02.197086 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.197901 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.199561 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.200006 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.200107 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:02.200545 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:02.200640 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:02.200709 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:02.200832 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 20:53:02.200905 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:53:02.201679 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:02.182013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38095 (local address 127.0.97.1:53035)
0929 20:53:02.195504 (+ 13491us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:02.195846 (+ 342us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:02.196057 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:02.196636 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:02.196639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:02.196648 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:53:02.196946 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:02.196955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.198019 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.198024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:02.199206 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:02.199217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.200175 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.200182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:02.200323 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:02.200382 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:53:02.200728 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:53:02.200739 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:53:02.200813 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 20:53:02.200943 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13230,"thread_start_us":13518,"threads_started":1}
I20260929 20:53:02.241428 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:02.241647 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:02.244499 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6c89bced60149259508590bf327aa0d: Sending DeleteTablet for 3 replicas of tablet 5fb04c9a08a847f5bcf4095eaaaff134
I20260929 20:53:02.245393 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fb04c9a08a847f5bcf4095eaaaff134 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34816
I20260929 20:53:02.245713 2041 tablet_replica.cc:330] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54: stopping tablet replica
I20260929 20:53:02.245673 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fb04c9a08a847f5bcf4095eaaaff134 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37124
I20260929 20:53:02.245931 2041 raft_consensus.cc:2229] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:02.245950 2040 tablet_replica.cc:330] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003: stopping tablet replica
I20260929 20:53:02.246138 2040 raft_consensus.cc:2229] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:53:02.246145 2041 raft_consensus.cc:2258] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:02.246511 2040 raft_consensus.cc:2258] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:02.247521 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fb04c9a08a847f5bcf4095eaaaff134 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57682
I20260929 20:53:02.247802 2039 tablet_replica.cc:330] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d: stopping tablet replica
I20260929 20:53:02.247977 2039 raft_consensus.cc:2229] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:02.248112 2041 ts_tablet_manager.cc:1905] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:02.248149 2039 raft_consensus.cc:2258] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:02.249296 2039 ts_tablet_manager.cc:1905] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:02.255259 2040 ts_tablet_manager.cc:1905] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:02.255715 2041 ts_tablet_manager.cc:1918] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:53:02.255940 2041 log.cc:1196] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/5fb04c9a08a847f5bcf4095eaaaff134
I20260929 20:53:02.256608 2041 ts_tablet_manager.cc:1939] T 5fb04c9a08a847f5bcf4095eaaaff134 P 4664b98e374042c3970dee6322790e54: Deleting consensus metadata
I20260929 20:53:02.257867 1496 catalog_manager.cc:4841] TS 4664b98e374042c3970dee6322790e54 (127.0.97.1:41841): tablet 5fb04c9a08a847f5bcf4095eaaaff134 (table test-table [id=0a4f852708314291a9dc5f4ef830e4ca]) successfully deleted
I20260929 20:53:02.261317 2040 ts_tablet_manager.cc:1918] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:53:02.261478 2040 log.cc:1196] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/5fb04c9a08a847f5bcf4095eaaaff134
I20260929 20:53:02.261914 2040 ts_tablet_manager.cc:1939] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3af7d71f8f7340c3b27d110f32891003: Deleting consensus metadata
I20260929 20:53:02.263093 1495 catalog_manager.cc:4841] TS 3af7d71f8f7340c3b27d110f32891003 (127.0.97.2:38095): tablet 5fb04c9a08a847f5bcf4095eaaaff134 (table test-table [id=0a4f852708314291a9dc5f4ef830e4ca]) successfully deleted
I20260929 20:53:02.265055 2039 ts_tablet_manager.cc:1918] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:53:02.265200 2039 log.cc:1196] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/5fb04c9a08a847f5bcf4095eaaaff134
I20260929 20:53:02.266443 2039 ts_tablet_manager.cc:1939] T 5fb04c9a08a847f5bcf4095eaaaff134 P 3baf8a4084544614af931f0ce0884e8d: Deleting consensus metadata
I20260929 20:53:02.267462 1495 catalog_manager.cc:4841] TS 3baf8a4084544614af931f0ce0884e8d (127.0.97.3:36873): tablet 5fb04c9a08a847f5bcf4095eaaaff134 (table test-table [id=0a4f852708314291a9dc5f4ef830e4ca]) successfully deleted
Sep 29 20:53:02 dist-test-slave-m9g1 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715181, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:53:02.286994 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:02.271821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34832 (local address 127.0.97.1:41841)
0929 20:53:02.272069 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:02.272076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:02.272630 (+ 554us) server_negotiation.cc:407] Connection header received
0929 20:53:02.272851 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:02.272857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:02.272941 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:02.273030 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:02.274817 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.275630 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.276849 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.277321 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.277430 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:02.281117 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:02.281131 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:02.281147 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:02.281182 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:02.284078 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:02.284755 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:02.284763 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:02.284765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:02.284930 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:02.285308 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:02.285313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:02.285316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:02.285769 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:02.286084 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:02.286455 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 20:53:02.286682 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260929 20:53:02.287811 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34832
I20260929 20:53:02.309620 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:02.292075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.0.97.62:39989)
0929 20:53:02.292268 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:02.292272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:02.296570 (+ 4298us) server_negotiation.cc:407] Connection header received
0929 20:53:02.296807 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:02.296813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:02.296898 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:02.296982 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:02.298807 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.299537 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.300708 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.301120 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.301240 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:02.304091 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:02.304096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:02.304099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:02.304135 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:02.306670 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:02.307353 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:02.307357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:02.307359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:02.307479 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:02.308258 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:02.308261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:02.308263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:02.308540 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:02.308784 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:02.309078 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 20:53:02.309302 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 20:53:02.310364 1508 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44400
I20260929 20:53:02.311934 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 1582
I20260929 20:53:02.325917 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:02.313109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34848 (local address 127.0.97.1:41841)
0929 20:53:02.313831 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:02.313841 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:02.314356 (+ 515us) server_negotiation.cc:407] Connection header received
0929 20:53:02.314685 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:02.314690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:02.314774 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:02.314835 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:02.316105 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.316820 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.317964 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.318449 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.318557 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:02.320869 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:02.320874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:02.320877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:02.320909 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:02.323561 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:02.324177 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:02.324181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:02.324183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:02.324298 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:02.324610 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:02.324619 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:02.324622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:02.324891 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:02.325108 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:02.325395 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 20:53:02.325607 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556}
W20260929 20:53:02.326450 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34848
W20260929 20:53:02.326973 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 1582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:53:02.327126 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 1582
I20260929 20:53:02.342036 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 1722
Sep 29 20:53:02 dist-test-slave-m9g1 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715181, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:53:02.355199 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:02.342750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37142 (local address 127.0.97.2:38095)
0929 20:53:02.342927 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:02.342932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:02.342950 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:53:02.343047 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:02.343050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:02.343117 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:02.343203 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:02.344237 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.344951 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.346067 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.346589 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.346681 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:02.350329 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:02.350344 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:02.350363 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:02.350396 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:02.352903 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:02.353486 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:02.353492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:02.353495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:02.353622 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:02.353890 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:02.353893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:02.353895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:02.354250 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:02.354516 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:02.354672 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 20:53:02.354881 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 20:53:02.355814 1852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37142
W20260929 20:53:02.356355 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 1722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:53:02.356498 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 1722
I20260929 20:53:02.374497 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 1861
Sep 29 20:53:02 dist-test-slave-m9g1 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715181, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:53:02.388609 2078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:02.375626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57696 (local address 127.0.97.3:36873)
0929 20:53:02.376131 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:02.376137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:02.376157 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:53:02.376220 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:02.376225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:02.376298 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:02.376398 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:02.377515 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.378236 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.379340 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:02.379725 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:02.379817 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:02.383646 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:02.383660 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:02.383676 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:02.383708 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:02.386468 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:02.387052 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:02.387059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:02.387062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:02.387174 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:02.387423 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:02.387426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:02.387427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:02.387754 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:02.387969 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:02.388138 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 20:53:02.388297 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":234,"threads_started":1}
W20260929 20:53:02.389283 1990 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57696
W20260929 20:53:02.389880 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 1861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:53:02.390009 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 1861
I20260929 20:53:02.403033 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 1479
I20260929 20:53:02.404091 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:02.515129 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 1479
2026-09-29T20:53:02Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4177 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:53:02 dist-test-slave-m9g1 krb5kdc[2082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:53:02 dist-test-slave-m9g1 krb5kdc[2082](info): set up 2 sockets
Sep 29 20:53:02 dist-test-slave-m9g1 krb5kdc[2082](info): commencing operation
krb5kdc: starting...
W20260929 20:53:03.498083 388 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:53:03 dist-test-slave-m9g1 krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:53:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:53:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:53:03.666445 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37599
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:03.977139 2095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:03.977461 2095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:03.977587 2095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:03.984897 2095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:53:03.985024 2095 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 20:53:03.985092 2095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:03.985173 2095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:03.985235 2095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:53:03.985316 2095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:53:03.995375 2095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37599
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:03.996654 2095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:03.997941 2095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:04.009267 2100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:04.009305 2101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:04.009217 2103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:04.010113 2095 server_base.cc:1034] running on GCE node
I20260929 20:53:04.010982 2095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:04.012370 2095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:04.013581 2095 hybrid_clock.cc:648] HybridClock initialized: now 1790715184013523 us; error 31 us; skew 500 ppm
I20260929 20:53:04.013885 2095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:04 dist-test-slave-m9g1 krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:04.022146 2095 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:53:04.023681 2095 webserver.cc:460] Webserver started at http://127.0.97.62:41897/ using document root <none> and password file <none>
I20260929 20:53:04.024370 2095 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:04.024456 2095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:04.024729 2095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:04.027667 2095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "9de46b11bfeb4eeba2a2093b3e1f713a"
format_stamp: "Formatted at 2026-09-29 20:53:04 on dist-test-slave-m9g1"
server_key: "11503846c93a4b15b0342ebc6d949371"
server_key_iv: "b94211b1ab594c3d555cc8c9a5eca7df"
server_key_version: "encryptionkey@0"
I20260929 20:53:04.028364 2095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "9de46b11bfeb4eeba2a2093b3e1f713a"
format_stamp: "Formatted at 2026-09-29 20:53:04 on dist-test-slave-m9g1"
server_key: "11503846c93a4b15b0342ebc6d949371"
server_key_iv: "b94211b1ab594c3d555cc8c9a5eca7df"
server_key_version: "encryptionkey@0"
I20260929 20:53:04.033444 2095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 20:53:04.036913 2110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:04.038302 2095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:53:04.038437 2095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "9de46b11bfeb4eeba2a2093b3e1f713a"
format_stamp: "Formatted at 2026-09-29 20:53:04 on dist-test-slave-m9g1"
server_key: "11503846c93a4b15b0342ebc6d949371"
server_key_iv: "b94211b1ab594c3d555cc8c9a5eca7df"
server_key_version: "encryptionkey@0"
I20260929 20:53:04.038648 2095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:04.056478 2095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:04.060775 2095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:04.061023 2095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:04.080817 2095 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37599
I20260929 20:53:04.080847 2171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37599 every 8 connection(s)
I20260929 20:53:04.082249 2095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:53:04.086051 2172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:04.092338 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 2095
I20260929 20:53:04.092530 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:53:04.093120 388 external_mini_cluster.cc:1428] Setting key 3b7a126ce310613f9a1e049647beb95b
I20260929 20:53:04.097242 2172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a: Bootstrap starting.
I20260929 20:53:04.101236 2172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:04.102533 2172 log.cc:824] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:04.105652 2172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a: No bootstrap required, opened a new log
Sep 29 20:53:04 dist-test-slave-m9g1 krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:04.114967 2172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de46b11bfeb4eeba2a2093b3e1f713a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37599 } }
I20260929 20:53:04.115454 2172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:04.115551 2172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de46b11bfeb4eeba2a2093b3e1f713a, State: Initialized, Role: FOLLOWER
I20260929 20:53:04.116242 2172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de46b11bfeb4eeba2a2093b3e1f713a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37599 } }
I20260929 20:53:04.116536 2172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:53:04.116674 2172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:53:04.116851 2172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:04.117187 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:04.095195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58570 (local address 127.0.97.62:37599)
0929 20:53:04.095990 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:04.096007 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:04.096058 (+ 51us) server_negotiation.cc:407] Connection header received
0929 20:53:04.098375 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:04.098405 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:04.098928 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:04.099440 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:04.100623 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:04.102629 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:04.104138 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:04.105122 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:04.105309 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:04.108827 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:04.108841 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:04.108861 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:04.108918 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:04.112787 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:04.113430 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:04.113440 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:04.113453 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:04.113659 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:04.113958 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:04.113963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:04.113966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:04.114705 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:04.114977 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:04.115264 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 20:53:04.115708 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":338,"threads_started":1}
I20260929 20:53:04.119716 2172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de46b11bfeb4eeba2a2093b3e1f713a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37599 } }
I20260929 20:53:04.120217 2172 leader_election.cc:302] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9de46b11bfeb4eeba2a2093b3e1f713a; no voters:
I20260929 20:53:04.120718 2172 leader_election.cc:288] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:53:04.120834 2177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:04.121821 2177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a [term 1 LEADER]: Becoming Leader. State: Replica: 9de46b11bfeb4eeba2a2093b3e1f713a, State: Running, Role: LEADER
I20260929 20:53:04.122377 2177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de46b11bfeb4eeba2a2093b3e1f713a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37599 } }
I20260929 20:53:04.122757 2172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:53:04.124763 2178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9de46b11bfeb4eeba2a2093b3e1f713a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de46b11bfeb4eeba2a2093b3e1f713a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37599 } } }
I20260929 20:53:04.124850 2179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9de46b11bfeb4eeba2a2093b3e1f713a. Latest consensus state: current_term: 1 leader_uuid: "9de46b11bfeb4eeba2a2093b3e1f713a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de46b11bfeb4eeba2a2093b3e1f713a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37599 } } }
I20260929 20:53:04.125073 2178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:04.125175 2179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:04.126009 2184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:53:04.132454 2184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:53:04.141407 2184 catalog_manager.cc:1347] Generated new cluster ID: 9e55b278accc49dc826288e8a5367894
I20260929 20:53:04.141541 2184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:53:04.154734 2184 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:53:04.156611 2184 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:53:04.164953 2184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a: Generated new TSK 0
I20260929 20:53:04.165817 2184 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:53:04.215389 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37599
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:04.543079 2198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:04.543375 2198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:04.543507 2198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:04.550714 2198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:04.550843 2198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:04.550966 2198 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 20:53:04.551025 2198 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:53:04.551133 2198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:53:04.560920 2198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:04.562263 2198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:04.563584 2198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:04.578326 2203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:04.578305 2206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:04.578508 2204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:04.579392 2198 server_base.cc:1034] running on GCE node
I20260929 20:53:04.580061 2198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:04.581023 2198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:04.582237 2198 hybrid_clock.cc:648] HybridClock initialized: now 1790715184582159 us; error 49 us; skew 500 ppm
I20260929 20:53:04.582544 2198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:04 dist-test-slave-m9g1 krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:04.585425 2198 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:53:04.586855 2198 webserver.cc:460] Webserver started at http://127.0.97.1:40287/ using document root <none> and password file <none>
I20260929 20:53:04.587513 2198 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:04.587635 2198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:04.587911 2198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:04.590899 2198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "a167d1b3dc124c49bf3e69dd639c2c63"
format_stamp: "Formatted at 2026-09-29 20:53:04 on dist-test-slave-m9g1"
server_key: "6e9fa343debdd157f96bb95f9d388e24"
server_key_iv: "ce755ad81765b604156e28ec83a2808a"
server_key_version: "encryptionkey@0"
I20260929 20:53:04.591672 2198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a167d1b3dc124c49bf3e69dd639c2c63"
format_stamp: "Formatted at 2026-09-29 20:53:04 on dist-test-slave-m9g1"
server_key: "6e9fa343debdd157f96bb95f9d388e24"
server_key_iv: "ce755ad81765b604156e28ec83a2808a"
server_key_version: "encryptionkey@0"
I20260929 20:53:04.596839 2198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 20:53:04.600286 2213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:04.601724 2198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:53:04.601904 2198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "a167d1b3dc124c49bf3e69dd639c2c63"
format_stamp: "Formatted at 2026-09-29 20:53:04 on dist-test-slave-m9g1"
server_key: "6e9fa343debdd157f96bb95f9d388e24"
server_key_iv: "ce755ad81765b604156e28ec83a2808a"
server_key_version: "encryptionkey@0"
I20260929 20:53:04.602102 2198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:04.615515 2198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:04.619282 2198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:04.619544 2198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:04.622133 2198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:04.622279 2198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 20:53:04.622442 2198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:04.622522 2198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:53:04 dist-test-slave-m9g1 krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:04.644876 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:04.622518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49941 (local address 127.0.97.62:37599)
0929 20:53:04.622989 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:04.622996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:04.624621 (+ 1625us) server_negotiation.cc:407] Connection header received
0929 20:53:04.626450 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:04.626457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:04.626544 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:04.626698 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:04.628708 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:04.629648 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:04.632498 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:04.632984 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:04.633107 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:04.638005 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:04.638012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:04.638015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:04.638056 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:04.641234 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:04.642177 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:04.642201 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:04.642205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:04.642365 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:04.642911 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:04.642916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:04.642920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:04.643299 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:04.643458 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:04.644216 (+ 758us) server_negotiation.cc:299] Negotiation successful
0929 20:53:04.644459 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":226,"threads_started":1}
I20260929 20:53:04.645675 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:04.623222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37599 (local address 127.0.97.1:49941)
0929 20:53:04.624096 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:04.624141 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:04.626118 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:04.626958 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:04.626973 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:04.627624 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:04.628519 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:04.628549 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:04.629831 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:04.629838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:04.632339 (+ 2501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:04.632355 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:04.633412 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:04.633417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:04.633657 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:04.634417 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:04.634440 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:04.637717 (+ 3277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:04.641386 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:04.641397 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:04.641421 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:04.642029 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:04.642489 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:04.642495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:04.642499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:04.642795 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:04.643425 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:04.643432 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:04.643846 (+ 414us) client_negotiation.cc:769] Sending connection context
0929 20:53:04.644106 (+ 260us) client_negotiation.cc:240] Negotiation successful
0929 20:53:04.644511 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":351,"threads_started":1}
I20260929 20:53:04.665632 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:04.651778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43685 (local address 127.0.97.62:37599)
0929 20:53:04.651962 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:04.651966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:04.652505 (+ 539us) server_negotiation.cc:407] Connection header received
0929 20:53:04.652674 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:04.652679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:04.652748 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:04.652827 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:04.653733 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:04.654580 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:04.656842 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:04.657417 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:04.657543 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:04.659750 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:04.659757 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:04.659762 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:04.659805 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:04.662912 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:04.663546 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:04.663552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:04.663556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:04.663685 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:04.664044 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:04.664135 (+ 91us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:04.664141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:04.664468 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:04.664629 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:04.665038 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 20:53:04.665272 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 20:53:04.665743 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:04.651857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37599 (local address 127.0.97.1:43685)
0929 20:53:04.652401 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:04.652419 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:04.652564 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:04.652866 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:04.652871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:04.653332 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:04.653600 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:04.653609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:04.654730 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:04.654736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:04.656680 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:04.656700 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:04.657779 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:04.657785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:04.657966 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:04.658724 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:04.658744 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:04.659565 (+ 821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:04.663048 (+ 3483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:04.663054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:04.663059 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:04.663437 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:04.663785 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:04.663790 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:04.663795 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:04.663919 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:04.664610 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:04.664615 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:04.664809 (+ 194us) client_negotiation.cc:769] Sending connection context
0929 20:53:04.664928 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 20:53:04.665257 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":304,"threads_started":1}
I20260929 20:53:04.670951 2198 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32949
I20260929 20:53:04.670979 2333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32949 every 8 connection(s)
I20260929 20:53:04.672171 2198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:53:04.674139 2334 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37599
I20260929 20:53:04.674401 2334 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:04.674609 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 2198
I20260929 20:53:04.674800 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:53:04.675236 2334 heartbeater.cc:511] Master 127.0.97.62:37599 requested a full tablet report, sending...
I20260929 20:53:04.675577 388 external_mini_cluster.cc:1428] Setting key 44b58969f497fb7dd3419375b712a40e
I20260929 20:53:04.677431 2127 ts_manager.cc:194] Registered new tserver with Master: a167d1b3dc124c49bf3e69dd639c2c63 (127.0.97.1:32949)
I20260929 20:53:04.678930 2127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49941
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:53:04.711203 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37599
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:05.023101 2338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:05.023378 2338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:05.023524 2338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:05.030714 2338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:05.030836 2338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:05.030961 2338 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 20:53:05.031033 2338 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:53:05.031137 2338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:53:05.040779 2338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:05.042121 2338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:05.043504 2338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:05.056406 2343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:05.056484 2344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:05.056459 2346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:05.056915 2338 server_base.cc:1034] running on GCE node
I20260929 20:53:05.057673 2338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:05.058668 2338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:05.059857 2338 hybrid_clock.cc:648] HybridClock initialized: now 1790715185059808 us; error 43 us; skew 500 ppm
I20260929 20:53:05.060151 2338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:05 dist-test-slave-m9g1 krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:05.063138 2338 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:53:05.064625 2338 webserver.cc:460] Webserver started at http://127.0.97.2:34955/ using document root <none> and password file <none>
I20260929 20:53:05.065315 2338 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:05.065418 2338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:05.065718 2338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:05.068759 2338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "a5ffffb3986541a7a50d7afa60d83e94"
format_stamp: "Formatted at 2026-09-29 20:53:05 on dist-test-slave-m9g1"
server_key: "4d882438d81de6b9cafed4844ebbbd73"
server_key_iv: "714fc6670560002244b750919fa1be82"
server_key_version: "encryptionkey@0"
I20260929 20:53:05.069568 2338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a5ffffb3986541a7a50d7afa60d83e94"
format_stamp: "Formatted at 2026-09-29 20:53:05 on dist-test-slave-m9g1"
server_key: "4d882438d81de6b9cafed4844ebbbd73"
server_key_iv: "714fc6670560002244b750919fa1be82"
server_key_version: "encryptionkey@0"
I20260929 20:53:05.075309 2338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 20:53:05.078864 2353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:05.080343 2338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:53:05.080497 2338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "a5ffffb3986541a7a50d7afa60d83e94"
format_stamp: "Formatted at 2026-09-29 20:53:05 on dist-test-slave-m9g1"
server_key: "4d882438d81de6b9cafed4844ebbbd73"
server_key_iv: "714fc6670560002244b750919fa1be82"
server_key_version: "encryptionkey@0"
I20260929 20:53:05.080688 2338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:05.100662 2338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:05.104823 2338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:05.105084 2338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:05.107858 2338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:05.107981 2338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:05.108098 2338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:05.108184 2338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:53:05 dist-test-slave-m9g1 krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:05.130864 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.108291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50489 (local address 127.0.97.62:37599)
0929 20:53:05.108481 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:05.108488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:05.110054 (+ 1566us) server_negotiation.cc:407] Connection header received
0929 20:53:05.112142 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:05.112148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:05.112236 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:05.112322 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:05.114573 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.115478 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.118386 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.118919 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.119052 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.124456 (+ 5404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:05.124461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:05.124464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:05.124499 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:05.127260 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:05.128260 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:05.128265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:05.128269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:05.128411 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:05.128954 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:05.128960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:05.128963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:05.129246 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:05.129379 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:05.130312 (+ 933us) server_negotiation.cc:299] Negotiation successful
0929 20:53:05.130540 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 20:53:05.131999 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.109040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37599 (local address 127.0.97.2:50489)
0929 20:53:05.109895 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:05.109939 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:05.111844 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:05.112653 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:05.112666 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:05.113453 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:05.114381 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.114415 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.115665 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.115671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.118166 (+ 2495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.118222 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.119184 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.119196 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.119527 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.120363 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:05.120419 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:05.124151 (+ 3732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:05.127392 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:05.127410 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:05.127434 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:05.128115 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:05.128527 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:05.128532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:05.128536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:05.128844 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:05.129370 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:05.129378 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:05.129832 (+ 454us) client_negotiation.cc:769] Sending connection context
0929 20:53:05.130211 (+ 379us) client_negotiation.cc:240] Negotiation successful
0929 20:53:05.130763 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":400,"threads_started":1}
I20260929 20:53:05.150749 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.137458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37143 (local address 127.0.97.62:37599)
0929 20:53:05.137595 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:05.137599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:05.138237 (+ 638us) server_negotiation.cc:407] Connection header received
0929 20:53:05.138415 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:05.138421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:05.138483 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:05.138561 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:05.139850 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.140825 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.142160 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.142778 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.142908 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.145218 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:05.145225 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:05.145230 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:05.145283 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:05.148266 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:05.148908 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:05.148913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:05.148916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:05.149029 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:05.149354 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:05.149358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:05.149361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:05.149675 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:05.149811 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:05.150208 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 20:53:05.150411 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 20:53:05.150942 2430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.137542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37599 (local address 127.0.97.2:37143)
0929 20:53:05.138123 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:05.138138 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:05.138303 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:05.138602 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:05.138607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:05.139238 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:05.139650 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.139667 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.140953 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.140957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.142040 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.142052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.143159 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.143165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.143347 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.144054 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:05.144074 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:05.145030 (+ 956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:05.148411 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:05.148417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:05.148421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:05.148797 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:05.149129 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:05.149134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:05.149138 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:05.149259 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:05.149792 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:05.149796 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:05.149972 (+ 176us) client_negotiation.cc:769] Sending connection context
0929 20:53:05.150086 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 20:53:05.150468 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":313,"threads_started":1}
I20260929 20:53:05.157320 2338 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34179
I20260929 20:53:05.157353 2472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34179 every 8 connection(s)
I20260929 20:53:05.158538 2338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:53:05.160362 2473 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37599
I20260929 20:53:05.160565 2473 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:05.161176 2473 heartbeater.cc:511] Master 127.0.97.62:37599 requested a full tablet report, sending...
I20260929 20:53:05.162510 2127 ts_manager.cc:194] Registered new tserver with Master: a5ffffb3986541a7a50d7afa60d83e94 (127.0.97.2:34179)
I20260929 20:53:05.163347 2127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50489
I20260929 20:53:05.166926 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 2338
I20260929 20:53:05.167050 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:53:05.167421 388 external_mini_cluster.cc:1428] Setting key 67a20e12f237cc93e0d4feae64919759
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:53:05.205055 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37599
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:05.519076 2477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:05.519383 2477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:05.519508 2477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:05.526880 2477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:05.527009 2477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:05.527127 2477 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 20:53:05.527190 2477 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:53:05.527293 2477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:53:05.536989 2477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:05.538362 2477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:05.539691 2477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:05.551940 2483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:05.552028 2482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:05.552002 2485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:05.552657 2477 server_base.cc:1034] running on GCE node
I20260929 20:53:05.553303 2477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:05.554248 2477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:05.555454 2477 hybrid_clock.cc:648] HybridClock initialized: now 1790715185555403 us; error 45 us; skew 500 ppm
I20260929 20:53:05.555760 2477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:05 dist-test-slave-m9g1 krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:05.558709 2477 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:53:05.560269 2477 webserver.cc:460] Webserver started at http://127.0.97.3:41767/ using document root <none> and password file <none>
I20260929 20:53:05.560947 2477 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:05.561051 2477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:05.561349 2477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:05.564273 2477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "2a5a067ca2b74b7e8af4dffbca74ab25"
format_stamp: "Formatted at 2026-09-29 20:53:05 on dist-test-slave-m9g1"
server_key: "f548cbb26d7713724ba569b8a7c3f1f4"
server_key_iv: "f14ca0d96670ee9759eadd2faad8a020"
server_key_version: "encryptionkey@0"
I20260929 20:53:05.565032 2477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2a5a067ca2b74b7e8af4dffbca74ab25"
format_stamp: "Formatted at 2026-09-29 20:53:05 on dist-test-slave-m9g1"
server_key: "f548cbb26d7713724ba569b8a7c3f1f4"
server_key_iv: "f14ca0d96670ee9759eadd2faad8a020"
server_key_version: "encryptionkey@0"
I20260929 20:53:05.570315 2477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 20:53:05.574061 2492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:05.575563 2477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:53:05.575726 2477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "2a5a067ca2b74b7e8af4dffbca74ab25"
format_stamp: "Formatted at 2026-09-29 20:53:05 on dist-test-slave-m9g1"
server_key: "f548cbb26d7713724ba569b8a7c3f1f4"
server_key_iv: "f14ca0d96670ee9759eadd2faad8a020"
server_key_version: "encryptionkey@0"
I20260929 20:53:05.575915 2477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:05.603636 2477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:05.608055 2477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:05.608325 2477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:05.611049 2477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:05.611150 2477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:05.611265 2477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:05.611341 2477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:53:05 dist-test-slave-m9g1 krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:05.632604 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.611628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48263 (local address 127.0.97.62:37599)
0929 20:53:05.611830 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:05.611836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:05.613329 (+ 1493us) server_negotiation.cc:407] Connection header received
0929 20:53:05.615727 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:05.615733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:05.615827 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:05.615940 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:05.618465 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.619403 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.621607 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.622127 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.622295 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.626107 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:05.626112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:05.626116 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:05.626149 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:05.629067 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:05.629993 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:05.630000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:05.630004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:05.630222 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:05.630822 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:05.630828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:05.630831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:05.631156 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:05.631280 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:05.632020 (+ 740us) server_negotiation.cc:299] Negotiation successful
0929 20:53:05.632240 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:53:05.633669 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.612269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37599 (local address 127.0.97.3:48263)
0929 20:53:05.613139 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:05.613188 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:05.615320 (+ 2132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:05.616272 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:05.616290 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:05.617189 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:05.618247 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.618280 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.619595 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.619599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.621466 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.621481 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.622409 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.622417 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.622722 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.623440 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:05.623462 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:05.625908 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:05.629206 (+ 3298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:05.629220 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:05.629243 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:05.629854 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:05.630376 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:05.630382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:05.630386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:05.630701 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:05.631294 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:05.631307 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:05.631660 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 20:53:05.631904 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 20:53:05.632457 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":381,"threads_started":1}
I20260929 20:53:05.656797 2477 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45621
I20260929 20:53:05.656831 2611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45621 every 8 connection(s)
I20260929 20:53:05.658162 2477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:53:05.660388 2612 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37599
I20260929 20:53:05.660652 2612 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:05.660921 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 2477
I20260929 20:53:05.661077 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:53:05.661526 2612 heartbeater.cc:511] Master 127.0.97.62:37599 requested a full tablet report, sending...
I20260929 20:53:05.661588 388 external_mini_cluster.cc:1428] Setting key df62e198475d3958618f43928de9dbde
I20260929 20:53:05.663044 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.648724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54541 (local address 127.0.97.62:37599)
0929 20:53:05.648905 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:05.648912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:05.649759 (+ 847us) server_negotiation.cc:407] Connection header received
0929 20:53:05.649955 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:05.649958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:05.650022 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:05.650113 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:05.651729 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.652666 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.654850 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.655243 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.655361 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.657305 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:05.657310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:05.657314 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:05.657346 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:05.660213 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:05.661035 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:05.661041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:05.661045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:05.661190 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:05.661537 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:05.661543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:05.661546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:05.661824 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:05.661955 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:05.662480 (+ 525us) server_negotiation.cc:299] Negotiation successful
0929 20:53:05.662711 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:53:05.663075 2591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.648831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37599 (local address 127.0.97.3:54541)
0929 20:53:05.649648 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:05.649669 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:05.649840 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:05.650155 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:05.650160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:05.650896 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:05.651563 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.651578 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.652795 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.652801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.654719 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.654733 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.655441 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.655450 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.655602 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.656217 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:05.656236 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:05.657109 (+ 873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:05.660348 (+ 3239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:05.660353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:05.660357 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:05.660926 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:05.661295 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:05.661300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:05.661304 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:05.661438 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:05.661936 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:05.661940 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:05.662131 (+ 191us) client_negotiation.cc:769] Sending connection context
0929 20:53:05.662325 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 20:53:05.662656 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":488,"threads_started":1}
I20260929 20:53:05.663071 2126 ts_manager.cc:194] Registered new tserver with Master: 2a5a067ca2b74b7e8af4dffbca74ab25 (127.0.97.3:45621)
I20260929 20:53:05.664655 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:53:05.664618 2126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48263
I20260929 20:53:05.681876 2334 heartbeater.cc:503] Master 127.0.97.62:37599 was elected leader, sending a full tablet report...
Sep 29 20:53:05 dist-test-slave-m9g1 krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:53:05 dist-test-slave-m9g1 krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:05.710938 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.697704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58586 (local address 127.0.97.62:37599)
0929 20:53:05.697938 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:05.697945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:05.698050 (+ 105us) server_negotiation.cc:407] Connection header received
0929 20:53:05.698267 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:05.698272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:05.698350 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:05.698437 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:05.699825 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.700572 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.701804 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.702300 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.702408 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.706070 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:05.706075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:05.706079 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:05.706114 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:05.708458 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:05.709049 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:05.709054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:05.709057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:05.709180 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:05.709546 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:05.709551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:05.709555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:05.709818 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:05.710042 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:05.710370 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 20:53:05.710583 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 20:53:05.715052 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:53:05.721624 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:53:05.745743 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.733678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.0.97.3:45621)
0929 20:53:05.736469 (+ 2791us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:05.736480 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:05.736513 (+ 33us) server_negotiation.cc:407] Connection header received
0929 20:53:05.736619 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:05.736627 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:05.736905 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:05.737923 (+ 1018us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:05.738295 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.740045 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.743378 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.744540 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.744706 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.744936 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:05.745266 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 20:53:05.745428 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2600,"thread_start_us":1845,"threads_started":1}
I20260929 20:53:05.745810 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.733336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45621 (local address 127.0.0.1:51006)
0929 20:53:05.734495 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:05.734537 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:05.734712 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:05.737167 (+ 2455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:05.737176 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:05.737216 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:05.737683 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.737695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.740895 (+ 3200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.740901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.743213 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.743226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.744781 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.744788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.745008 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.745072 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 20:53:05.745215 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 20:53:05.745416 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"thread_start_us":736,"threads_started":1}
I20260929 20:53:05.749223 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.734956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34179 (local address 127.0.0.1:46854)
0929 20:53:05.741630 (+ 6674us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:05.741650 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:05.741780 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:05.742683 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:05.742687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:05.742711 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:05.743152 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.743165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.744639 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.744642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.746698 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.746714 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.748393 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.748399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.748564 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.748595 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:53:05.748682 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 20:53:05.748805 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6554,"spinlock_wait_cycles":768,"thread_start_us":6422,"threads_started":1}
I20260929 20:53:05.749704 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.734315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57422 (local address 127.0.97.1:32949)
0929 20:53:05.740950 (+ 6635us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:05.740960 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:05.740993 (+ 33us) server_negotiation.cc:407] Connection header received
0929 20:53:05.741091 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:05.741098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:05.741370 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:05.741627 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:05.742362 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.743914 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.746249 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.748573 (+ 2324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.748699 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.748839 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:05.749244 (+ 405us) server_negotiation.cc:299] Negotiation successful
0929 20:53:05.749404 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6465,"thread_start_us":6338,"threads_started":1}
I20260929 20:53:05.749799 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.733552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32949 (local address 127.0.0.1:57422)
0929 20:53:05.740636 (+ 7084us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:05.740657 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:05.740760 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:05.741644 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:05.741652 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:05.741676 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:05.742148 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.742160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.744056 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.744061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.746050 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.746063 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.748954 (+ 2891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.748960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.749075 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.749100 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 20:53:05.749182 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 20:53:05.749425 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6946,"mutex_wait_us":4,"spinlock_wait_cycles":14208,"thread_start_us":6779,"threads_started":1}
I20260929 20:53:05.749876 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.735455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46854 (local address 127.0.97.2:34179)
0929 20:53:05.741234 (+ 5779us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:05.741242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:05.741890 (+ 648us) server_negotiation.cc:407] Connection header received
0929 20:53:05.741999 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:05.742007 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:05.742275 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:05.742466 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:05.743344 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.744495 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.746886 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.748187 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.748321 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.748529 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:05.749437 (+ 908us) server_negotiation.cc:299] Negotiation successful
0929 20:53:05.749633 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5632,"thread_start_us":5541,"threads_started":1}
I20260929 20:53:05.749971 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 6673f54f330f4085958e63582aed4389 (DEFAULT_TABLE table=test-table [id=11a6bd8abb7d4c30a7258108f8daabfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:05.751537 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6673f54f330f4085958e63582aed4389. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:05.753355 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 6673f54f330f4085958e63582aed4389 (DEFAULT_TABLE table=test-table [id=11a6bd8abb7d4c30a7258108f8daabfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:05.753363 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 6673f54f330f4085958e63582aed4389 (DEFAULT_TABLE table=test-table [id=11a6bd8abb7d4c30a7258108f8daabfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:05.754503 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6673f54f330f4085958e63582aed4389. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:05.754631 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6673f54f330f4085958e63582aed4389. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:05.766502 2631 tablet_bootstrap.cc:492] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63: Bootstrap starting.
I20260929 20:53:05.768328 2633 tablet_bootstrap.cc:492] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94: Bootstrap starting.
I20260929 20:53:05.769826 2631 tablet_bootstrap.cc:654] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:05.769940 2632 tablet_bootstrap.cc:492] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25: Bootstrap starting.
I20260929 20:53:05.771198 2631 log.cc:824] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:05.772025 2633 tablet_bootstrap.cc:654] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:05.772993 2632 tablet_bootstrap.cc:654] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:05.773123 2633 log.cc:824] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:05.774062 2632 log.cc:824] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:05.775358 2631 tablet_bootstrap.cc:492] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63: No bootstrap required, opened a new log
I20260929 20:53:05.775626 2631 ts_tablet_manager.cc:1397] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 20:53:05.776000 2633 tablet_bootstrap.cc:492] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94: No bootstrap required, opened a new log
I20260929 20:53:05.776262 2633 ts_tablet_manager.cc:1397] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 20:53:05.776978 2632 tablet_bootstrap.cc:492] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25: No bootstrap required, opened a new log
I20260929 20:53:05.777220 2632 ts_tablet_manager.cc:1397] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260929 20:53:05.782950 2631 raft_consensus.cc:348] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } } peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } }
I20260929 20:53:05.783339 2631 raft_consensus.cc:374] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:05.783429 2631 raft_consensus.cc:729] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a167d1b3dc124c49bf3e69dd639c2c63, State: Initialized, Role: FOLLOWER
I20260929 20:53:05.784072 2631 consensus_queue.cc:260] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } } peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } }
I20260929 20:53:05.785917 2631 ts_tablet_manager.cc:1428] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260929 20:53:05.786600 2633 raft_consensus.cc:348] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } } peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } }
I20260929 20:53:05.787038 2633 raft_consensus.cc:374] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:05.787140 2633 raft_consensus.cc:729] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ffffb3986541a7a50d7afa60d83e94, State: Initialized, Role: FOLLOWER
I20260929 20:53:05.787818 2633 consensus_queue.cc:260] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } } peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } }
I20260929 20:53:05.790119 2632 raft_consensus.cc:348] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } } peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } }
I20260929 20:53:05.790640 2632 raft_consensus.cc:374] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:05.790737 2632 raft_consensus.cc:729] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a5a067ca2b74b7e8af4dffbca74ab25, State: Initialized, Role: FOLLOWER
I20260929 20:53:05.791384 2632 consensus_queue.cc:260] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } } peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } }
I20260929 20:53:05.791965 2473 heartbeater.cc:503] Master 127.0.97.62:37599 was elected leader, sending a full tablet report...
I20260929 20:53:05.792575 2612 heartbeater.cc:503] Master 127.0.97.62:37599 was elected leader, sending a full tablet report...
I20260929 20:53:05.792709 2633 ts_tablet_manager.cc:1428] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94: Time spent starting tablet: real 0.016s user 0.008s sys 0.009s
I20260929 20:53:05.792953 2632 ts_tablet_manager.cc:1428] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260929 20:53:05.891599 2637 raft_consensus.cc:482] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:53:05.891867 2637 raft_consensus.cc:504] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } } peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } }
I20260929 20:53:05.893544 2637 leader_election.cc:288] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a5a067ca2b74b7e8af4dffbca74ab25 (127.0.97.3:45621), a5ffffb3986541a7a50d7afa60d83e94 (127.0.97.2:34179)
I20260929 20:53:05.899675 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.893873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47703 (local address 127.0.97.3:45621)
0929 20:53:05.894128 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:05.894135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:05.894546 (+ 411us) server_negotiation.cc:407] Connection header received
0929 20:53:05.894737 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:05.894743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:05.894822 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:05.894908 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:05.895581 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.896322 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.898025 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.898816 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.898938 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.899038 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:05.899312 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 20:53:05.899433 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:53:05.899749 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.893819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45621 (local address 127.0.97.1:47703)
0929 20:53:05.894399 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:05.894415 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:05.894569 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:05.895076 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:05.895082 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:05.895112 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:05.895436 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.895444 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.896457 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.896461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.897867 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.897878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.899025 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.899031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.899168 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.899191 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:53:05.899276 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 20:53:05.899416 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":274,"threads_started":1}
I20260929 20:53:05.900067 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.894806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36273 (local address 127.0.97.2:34179)
0929 20:53:05.894988 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:05.894993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:05.895279 (+ 286us) server_negotiation.cc:407] Connection header received
0929 20:53:05.895439 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:05.895445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:05.895511 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:05.895583 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:05.896092 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.896791 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.898473 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.899211 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.899327 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.899424 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:05.899701 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 20:53:05.899827 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:53:05.900149 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.894559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34179 (local address 127.0.97.1:36273)
0929 20:53:05.895178 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:05.895192 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:05.895313 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:05.895612 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:05.895616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:05.895647 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:05.895951 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.895966 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.896915 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.896920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.898314 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.898330 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.899424 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.899430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.899562 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.899583 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 20:53:05.899661 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 20:53:05.899780 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":4,"thread_start_us":317,"threads_started":1}
I20260929 20:53:05.900735 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6673f54f330f4085958e63582aed4389" candidate_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" is_pre_election: true
I20260929 20:53:05.901120 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6673f54f330f4085958e63582aed4389" candidate_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ffffb3986541a7a50d7afa60d83e94" is_pre_election: true
I20260929 20:53:05.901333 2563 raft_consensus.cc:2454] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a167d1b3dc124c49bf3e69dd639c2c63 in term 0.
I20260929 20:53:05.901647 2423 raft_consensus.cc:2454] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a167d1b3dc124c49bf3e69dd639c2c63 in term 0.
I20260929 20:53:05.902024 2214 leader_election.cc:302] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a5a067ca2b74b7e8af4dffbca74ab25, a167d1b3dc124c49bf3e69dd639c2c63; no voters:
I20260929 20:53:05.902413 2637 raft_consensus.cc:2781] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:53:05.902582 2637 raft_consensus.cc:482] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:53:05.902679 2637 raft_consensus.cc:3037] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:05.904793 2637 raft_consensus.cc:504] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } } peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } }
I20260929 20:53:05.905366 2637 leader_election.cc:288] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [CANDIDATE]: Term 1 election: Requested vote from peers 2a5a067ca2b74b7e8af4dffbca74ab25 (127.0.97.3:45621), a5ffffb3986541a7a50d7afa60d83e94 (127.0.97.2:34179)
I20260929 20:53:05.905763 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6673f54f330f4085958e63582aed4389" candidate_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25"
I20260929 20:53:05.905829 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6673f54f330f4085958e63582aed4389" candidate_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ffffb3986541a7a50d7afa60d83e94"
I20260929 20:53:05.905938 2563 raft_consensus.cc:3037] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:05.906034 2423 raft_consensus.cc:3037] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:05.908150 2563 raft_consensus.cc:2454] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a167d1b3dc124c49bf3e69dd639c2c63 in term 1.
I20260929 20:53:05.908228 2423 raft_consensus.cc:2454] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a167d1b3dc124c49bf3e69dd639c2c63 in term 1.
I20260929 20:53:05.908514 2214 leader_election.cc:302] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a5a067ca2b74b7e8af4dffbca74ab25, a167d1b3dc124c49bf3e69dd639c2c63; no voters:
I20260929 20:53:05.908826 2637 raft_consensus.cc:2781] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:05.909229 2637 raft_consensus.cc:686] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 LEADER]: Becoming Leader. State: Replica: a167d1b3dc124c49bf3e69dd639c2c63, State: Running, Role: LEADER
W20260929 20:53:05.909785 2613 tablet.cc:2369] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:53:05.909749 2637 consensus_queue.cc:237] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } } peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } }
W20260929 20:53:05.910638 2474 tablet.cc:2369] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:53:05.914063 2126 catalog_manager.cc:5495] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 reported cstate change: term changed from 0 to 1, leader changed from <none> to a167d1b3dc124c49bf3e69dd639c2c63 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } health_report { overall_health: UNKNOWN } } }
W20260929 20:53:05.924355 2335 tablet.cc:2369] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 20:53:05 dist-test-slave-m9g1 krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:05.965088 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.952547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.0.97.62:37599)
0929 20:53:05.952711 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:05.952715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:05.953187 (+ 472us) server_negotiation.cc:407] Connection header received
0929 20:53:05.953348 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:05.953353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:05.953428 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:05.953509 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:05.954683 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.955397 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.956592 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.956996 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.957099 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.960236 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:05.960240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:05.960244 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:05.960271 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:05.962640 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:05.963272 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:05.963277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:05.963279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:05.963405 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:05.963719 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:05.963723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:05.963727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:05.963973 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:05.964089 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:05.964501 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 20:53:05.964793 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":48}
I20260929 20:53:05.965117 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:05.952574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37599 (local address 127.0.0.1:58594)
0929 20:53:05.953081 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:05.953097 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:05.953243 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:05.953545 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:05.953550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:05.954278 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:05.954546 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.954556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.955517 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.955522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.956465 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:05.956476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:05.957160 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:05.957170 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:05.957311 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:05.957920 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:05.957937 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:05.960067 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:05.962753 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:05.962766 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:05.962774 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:05.963169 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:05.963498 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:05.963502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:05.963506 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:05.963628 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:05.964085 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:05.964094 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:05.964264 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 20:53:05.964403 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 20:53:05.964663 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":274,"threads_started":1}
I20260929 20:53:05.970085 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58594:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 20:53:05.971148 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:53:05.977471 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5ed4c7de0b4e6d9bdd76445c50020e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a6332b2f648a47ca82db3a5d47741541]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:53:05.977658 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5ed4c7de0b4e6d9bdd76445c50020e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a6332b2f648a47ca82db3a5d47741541]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:53:05.977522 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5ed4c7de0b4e6d9bdd76445c50020e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a6332b2f648a47ca82db3a5d47741541]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:53:05.978461 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5ed4c7de0b4e6d9bdd76445c50020e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:05.978467 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5ed4c7de0b4e6d9bdd76445c50020e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:05.978467 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5ed4c7de0b4e6d9bdd76445c50020e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:05.984423 2631 tablet_bootstrap.cc:492] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a167d1b3dc124c49bf3e69dd639c2c63: Bootstrap starting.
I20260929 20:53:05.985308 2633 tablet_bootstrap.cc:492] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a5ffffb3986541a7a50d7afa60d83e94: Bootstrap starting.
I20260929 20:53:05.986121 2631 tablet_bootstrap.cc:654] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a167d1b3dc124c49bf3e69dd639c2c63: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:05.986905 2633 tablet_bootstrap.cc:654] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a5ffffb3986541a7a50d7afa60d83e94: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:05.987859 2632 tablet_bootstrap.cc:492] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25: Bootstrap starting.
I20260929 20:53:05.988269 2631 tablet_bootstrap.cc:492] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a167d1b3dc124c49bf3e69dd639c2c63: No bootstrap required, opened a new log
I20260929 20:53:05.988430 2631 ts_tablet_manager.cc:1397] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a167d1b3dc124c49bf3e69dd639c2c63: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 20:53:05.989183 2631 raft_consensus.cc:348] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:05.989416 2632 tablet_bootstrap.cc:654] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:05.989431 2631 raft_consensus.cc:374] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:05.989513 2631 raft_consensus.cc:729] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a167d1b3dc124c49bf3e69dd639c2c63, State: Initialized, Role: FOLLOWER
I20260929 20:53:05.989740 2631 consensus_queue.cc:260] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a167d1b3dc124c49bf3e69dd639c2c63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:05.990341 2631 ts_tablet_manager.cc:1428] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a167d1b3dc124c49bf3e69dd639c2c63: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 20:53:05.990206 2637 tablet_replica.cc:439] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a167d1b3dc124c49bf3e69dd639c2c63: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } } }
I20260929 20:53:05.990504 2637 tablet_replica.cc:442] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a167d1b3dc124c49bf3e69dd639c2c63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:05.991899 2633 tablet_bootstrap.cc:492] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a5ffffb3986541a7a50d7afa60d83e94: No bootstrap required, opened a new log
I20260929 20:53:05.992017 2632 tablet_bootstrap.cc:492] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25: No bootstrap required, opened a new log
I20260929 20:53:05.992031 2633 ts_tablet_manager.cc:1397] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a5ffffb3986541a7a50d7afa60d83e94: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260929 20:53:05.992159 2632 ts_tablet_manager.cc:1397] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260929 20:53:05.992722 2633 raft_consensus.cc:348] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:05.992905 2633 raft_consensus.cc:374] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:05.992998 2633 raft_consensus.cc:729] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ffffb3986541a7a50d7afa60d83e94, State: Initialized, Role: FOLLOWER
I20260929 20:53:05.992854 2632 raft_consensus.cc:348] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:05.993088 2632 raft_consensus.cc:374] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:05.993171 2632 raft_consensus.cc:729] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a5a067ca2b74b7e8af4dffbca74ab25, State: Initialized, Role: FOLLOWER
I20260929 20:53:05.993217 2633 consensus_queue.cc:260] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a5ffffb3986541a7a50d7afa60d83e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:05.993376 2632 consensus_queue.cc:260] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:05.993871 2632 ts_tablet_manager.cc:1428] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 20:53:05.993683 2638 tablet_replica.cc:439] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a5ffffb3986541a7a50d7afa60d83e94: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } } }
I20260929 20:53:05.994045 2638 tablet_replica.cc:442] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a5ffffb3986541a7a50d7afa60d83e94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:05.994509 2633 ts_tablet_manager.cc:1428] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a5ffffb3986541a7a50d7afa60d83e94: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:53:05.994349 2639 tablet_replica.cc:439] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } } }
I20260929 20:53:05.994688 2639 tablet_replica.cc:442] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:06.051946 2639 raft_consensus.cc:482] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:53:06.052196 2639 raft_consensus.cc:504] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:06.053784 2639 leader_election.cc:288] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a167d1b3dc124c49bf3e69dd639c2c63 (127.0.97.1:32949), a5ffffb3986541a7a50d7afa60d83e94 (127.0.97.2:34179)
I20260929 20:53:06.059691 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:06.054290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50347 (local address 127.0.97.1:32949)
0929 20:53:06.054476 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:06.054482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:06.054776 (+ 294us) server_negotiation.cc:407] Connection header received
0929 20:53:06.054858 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:06.054863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:06.054956 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:06.055046 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:06.055408 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.056251 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.058001 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.058822 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.058949 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:06.059052 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:06.059311 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 20:53:06.059453 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:53:06.059695 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:06.054078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32949 (local address 127.0.97.3:50347)
0929 20:53:06.054461 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:06.054480 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:06.054640 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:06.055088 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:06.055092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:06.055115 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:06.055267 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:06.055274 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.056412 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.056416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:06.057838 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:06.057852 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.058984 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.058990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:06.059170 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:06.059194 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:53:06.059274 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 20:53:06.059415 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260929 20:53:06.060118 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:06.054231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45775 (local address 127.0.97.2:34179)
0929 20:53:06.054655 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:06.054661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:06.054997 (+ 336us) server_negotiation.cc:407] Connection header received
0929 20:53:06.055163 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:06.055168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:06.055240 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:06.055306 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:06.055796 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.056763 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.058478 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.059240 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.059362 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:06.059451 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:06.059737 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 20:53:06.059858 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311}
I20260929 20:53:06.060132 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:06.054118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34179 (local address 127.0.97.3:45775)
0929 20:53:06.054901 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:06.054915 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:06.055024 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:06.055345 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:06.055349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:06.055372 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:06.055653 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:06.055661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.056918 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.056922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:06.058330 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:06.058344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.059440 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.059446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:06.059592 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:06.059615 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:53:06.059695 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 20:53:06.059817 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":539,"threads_started":1}
I20260929 20:53:06.060776 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5ed4c7de0b4e6d9bdd76445c50020e" candidate_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" is_pre_election: true
I20260929 20:53:06.061190 2288 raft_consensus.cc:2454] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a5a067ca2b74b7e8af4dffbca74ab25 in term 0.
I20260929 20:53:06.061400 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5ed4c7de0b4e6d9bdd76445c50020e" candidate_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ffffb3986541a7a50d7afa60d83e94" is_pre_election: true
I20260929 20:53:06.061640 2423 raft_consensus.cc:2454] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a5a067ca2b74b7e8af4dffbca74ab25 in term 0.
I20260929 20:53:06.061852 2494 leader_election.cc:302] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a5a067ca2b74b7e8af4dffbca74ab25, a167d1b3dc124c49bf3e69dd639c2c63; no voters:
I20260929 20:53:06.062309 2639 raft_consensus.cc:2781] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:53:06.062469 2639 raft_consensus.cc:482] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:53:06.062565 2639 raft_consensus.cc:3037] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:06.064596 2639 raft_consensus.cc:504] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:06.065126 2639 leader_election.cc:288] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 [CANDIDATE]: Term 1 election: Requested vote from peers a167d1b3dc124c49bf3e69dd639c2c63 (127.0.97.1:32949), a5ffffb3986541a7a50d7afa60d83e94 (127.0.97.2:34179)
I20260929 20:53:06.065611 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5ed4c7de0b4e6d9bdd76445c50020e" candidate_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a167d1b3dc124c49bf3e69dd639c2c63"
I20260929 20:53:06.065626 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5ed4c7de0b4e6d9bdd76445c50020e" candidate_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ffffb3986541a7a50d7afa60d83e94"
I20260929 20:53:06.065781 2288 raft_consensus.cc:3037] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:06.065812 2423 raft_consensus.cc:3037] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:06.067740 2423 raft_consensus.cc:2454] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a5ffffb3986541a7a50d7afa60d83e94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a5a067ca2b74b7e8af4dffbca74ab25 in term 1.
I20260929 20:53:06.068032 2288 raft_consensus.cc:2454] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a5a067ca2b74b7e8af4dffbca74ab25 in term 1.
I20260929 20:53:06.068147 2493 leader_election.cc:302] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a5a067ca2b74b7e8af4dffbca74ab25, a5ffffb3986541a7a50d7afa60d83e94; no voters:
I20260929 20:53:06.068486 2639 raft_consensus.cc:2781] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:06.068915 2639 raft_consensus.cc:686] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 LEADER]: Becoming Leader. State: Replica: 2a5a067ca2b74b7e8af4dffbca74ab25, State: Running, Role: LEADER
I20260929 20:53:06.069392 2639 consensus_queue.cc:237] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:06.072639 2654 tablet_replica.cc:439] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25: TxnStatusTablet state changed. Reason: New leader 2a5a067ca2b74b7e8af4dffbca74ab25. Latest consensus state: current_term: 1 leader_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } } }
I20260929 20:53:06.072939 2654 tablet_replica.cc:442] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:53:06.073666 2656 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 20:53:06.073920 2656 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 20:53:06.074697 2125 catalog_manager.cc:5495] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a5a067ca2b74b7e8af4dffbca74ab25 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } health_report { overall_health: HEALTHY } } }
I20260929 20:53:06.114321 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:06.108173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45621 (local address 127.0.0.1:51020)
0929 20:53:06.108393 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:06.108409 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:06.108543 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:06.109014 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:06.109019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:06.109030 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:53:06.109208 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:06.109216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.110658 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.110661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:06.111639 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:06.111650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.112597 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.112602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:06.112750 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:06.112807 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:53:06.113829 (+ 1022us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:53:06.113847 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 20:53:06.113918 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 20:53:06.114033 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 20:53:06.114382 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:06.108133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51020 (local address 127.0.97.3:45621)
0929 20:53:06.108291 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:06.108297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:06.108473 (+ 176us) server_negotiation.cc:407] Connection header received
0929 20:53:06.108751 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:06.108755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:06.108844 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:06.108964 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:06.109363 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.110548 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.111773 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.112421 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.112562 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:06.112994 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:06.113714 (+ 720us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:06.113868 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:06.113965 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 20:53:06.114051 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 20:53:06.122459 2288 raft_consensus.cc:1261] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 FOLLOWER]: Refusing update from remote peer 2a5a067ca2b74b7e8af4dffbca74ab25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:06.122618 2423 raft_consensus.cc:1261] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a5ffffb3986541a7a50d7afa60d83e94 [term 1 FOLLOWER]: Refusing update from remote peer 2a5a067ca2b74b7e8af4dffbca74ab25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:06.123430 2654 consensus_queue.cc:1035] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:06.123708 2639 consensus_queue.cc:1035] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:06.134789 2637 tablet_replica.cc:439] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a167d1b3dc124c49bf3e69dd639c2c63: TxnStatusTablet state changed. Reason: New leader 2a5a067ca2b74b7e8af4dffbca74ab25. Latest consensus state: current_term: 1 leader_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } } }
I20260929 20:53:06.135106 2637 tablet_replica.cc:442] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a167d1b3dc124c49bf3e69dd639c2c63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:06.138077 2638 tablet_replica.cc:439] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a5ffffb3986541a7a50d7afa60d83e94: TxnStatusTablet state changed. Reason: New leader 2a5a067ca2b74b7e8af4dffbca74ab25. Latest consensus state: current_term: 1 leader_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } } }
I20260929 20:53:06.138370 2638 tablet_replica.cc:442] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a5ffffb3986541a7a50d7afa60d83e94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:06.139124 2659 mvcc.cc:205] Tried to move back new op lower bound from 7334769402348630016 to 7334769402144931840. Current Snapshot: MvccSnapshot[applied={T|T < 7334769402348630016}]
I20260929 20:53:06.146687 2639 tablet_replica.cc:439] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } } }
I20260929 20:53:06.147063 2639 tablet_replica.cc:442] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:53:06.150418 2639 tablet_replica.cc:439] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } } }
I20260929 20:53:06.150749 2639 tablet_replica.cc:442] T 6a5ed4c7de0b4e6d9bdd76445c50020e P 2a5a067ca2b74b7e8af4dffbca74ab25: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:53:06.154109 2638 tablet_replica.cc:439] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a5ffffb3986541a7a50d7afa60d83e94: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } } }
I20260929 20:53:06.154400 2638 tablet_replica.cc:442] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a5ffffb3986541a7a50d7afa60d83e94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:06.159390 2637 tablet_replica.cc:439] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a167d1b3dc124c49bf3e69dd639c2c63: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } } }
I20260929 20:53:06.159672 2637 tablet_replica.cc:442] T 6a5ed4c7de0b4e6d9bdd76445c50020e P a167d1b3dc124c49bf3e69dd639c2c63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:06.167968 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:06.161432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57434 (local address 127.0.97.1:32949)
0929 20:53:06.161609 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:06.161615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:06.161633 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:53:06.161722 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:06.161728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:06.161836 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:06.161984 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:06.162409 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.163521 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.165432 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.166014 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.166128 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:06.166690 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:06.167348 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:06.167481 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:06.167589 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 20:53:06.167673 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:53:06.185911 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:06.179832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44269 (local address 127.0.97.3:45621)
0929 20:53:06.180135 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:06.180140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:06.180529 (+ 389us) server_negotiation.cc:407] Connection header received
0929 20:53:06.180740 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:06.180744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:06.180811 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:06.180881 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:06.181442 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.182158 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.184234 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.184679 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.184787 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:06.185305 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:06.185412 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:06.185483 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:06.185613 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 20:53:06.185673 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260929 20:53:06.186117 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:06.179738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45621 (local address 127.0.97.1:44269)
0929 20:53:06.180418 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:06.180437 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:06.180600 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:06.180922 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:06.180927 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:06.180940 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:53:06.181295 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:06.181307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.182286 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.182292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:06.184084 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:06.184098 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.184846 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.184853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:06.185027 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:06.185099 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:53:06.185504 (+ 405us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:53:06.185517 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 20:53:06.185593 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 20:53:06.185734 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":405,"threads_started":1}
I20260929 20:53:06.196121 2423 raft_consensus.cc:1261] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94 [term 1 FOLLOWER]: Refusing update from remote peer a167d1b3dc124c49bf3e69dd639c2c63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:06.196507 2563 raft_consensus.cc:1261] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 FOLLOWER]: Refusing update from remote peer a167d1b3dc124c49bf3e69dd639c2c63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:06.196744 2642 consensus_queue.cc:1035] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:06.197139 2637 consensus_queue.cc:1035] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:06.203115 2661 mvcc.cc:205] Tried to move back new op lower bound from 7334769402653872128 to 7334769401491087360. Current Snapshot: MvccSnapshot[applied={T|T < 7334769402653872128}]
I20260929 20:53:06.230302 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:06.224392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49921 (local address 127.0.97.1:32949)
0929 20:53:06.224561 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:06.224567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:06.224999 (+ 432us) server_negotiation.cc:407] Connection header received
0929 20:53:06.225187 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:06.225192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:06.225272 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:06.225370 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:06.225912 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.226965 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.228364 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.228906 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.229023 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:06.229562 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:06.229686 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:06.229775 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:06.229916 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 20:53:06.229992 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 20:53:06.230460 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:06.224327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32949 (local address 127.0.97.3:49921)
0929 20:53:06.224895 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:06.224913 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:06.225059 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:06.225385 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:06.225390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:06.225402 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:53:06.225767 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:06.225778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.227085 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.227090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:06.228232 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:06.228243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.229082 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.229089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:06.229278 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:06.229374 (+ 96us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:53:06.229789 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:53:06.229804 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 20:53:06.229888 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 20:53:06.230028 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":244,"threads_started":1}
I20260929 20:53:06.271615 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:06.271911 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:06.276103 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a: Sending DeleteTablet for 3 replicas of tablet 6673f54f330f4085958e63582aed4389
I20260929 20:53:06.277266 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet 6673f54f330f4085958e63582aed4389 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51006
I20260929 20:53:06.277325 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 6673f54f330f4085958e63582aed4389 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57422
I20260929 20:53:06.278486 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet 6673f54f330f4085958e63582aed4389 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46854
I20260929 20:53:06.286878 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:53:06.287631 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:53:06.289234 2679 tablet_replica.cc:330] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63: stopping tablet replica
I20260929 20:53:06.289778 2680 tablet_replica.cc:330] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25: stopping tablet replica
I20260929 20:53:06.290408 2680 raft_consensus.cc:2229] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:06.290930 2679 raft_consensus.cc:2229] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:53:06.291649 2679 raft_consensus.cc:2258] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:06.292001 2680 raft_consensus.cc:2258] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:06.296332 2402 tablet_service.cc:1467] Processing CreateTablet for tablet 171394b0f761452bb222f1240d18e8ae (DEFAULT_TABLE table=test-table [id=8a57ace708544d94aee35dd6dc341ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:06.296454 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 171394b0f761452bb222f1240d18e8ae (DEFAULT_TABLE table=test-table [id=8a57ace708544d94aee35dd6dc341ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:06.296792 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171394b0f761452bb222f1240d18e8ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:06.296868 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171394b0f761452bb222f1240d18e8ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:06.302454 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 171394b0f761452bb222f1240d18e8ae (DEFAULT_TABLE table=test-table [id=8a57ace708544d94aee35dd6dc341ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:06.302963 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171394b0f761452bb222f1240d18e8ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:06.309556 2632 tablet_bootstrap.cc:492] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25: Bootstrap starting.
I20260929 20:53:06.310845 2633 tablet_bootstrap.cc:492] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94: Bootstrap starting.
I20260929 20:53:06.311317 2632 tablet_bootstrap.cc:654] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:06.312402 2633 tablet_bootstrap.cc:654] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:06.313031 2631 tablet_bootstrap.cc:492] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63: Bootstrap starting.
I20260929 20:53:06.314807 2631 tablet_bootstrap.cc:654] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:06.317523 2680 ts_tablet_manager.cc:1905] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:06.318110 2679 ts_tablet_manager.cc:1905] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:06.329790 2632 tablet_bootstrap.cc:492] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25: No bootstrap required, opened a new log
I20260929 20:53:06.329993 2632 ts_tablet_manager.cc:1397] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.005s
I20260929 20:53:06.330829 2632 raft_consensus.cc:348] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:06.330977 2633 tablet_bootstrap.cc:492] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94: No bootstrap required, opened a new log
I20260929 20:53:06.331084 2632 raft_consensus.cc:374] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:06.331167 2632 raft_consensus.cc:729] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a5a067ca2b74b7e8af4dffbca74ab25, State: Initialized, Role: FOLLOWER
I20260929 20:53:06.331367 2633 ts_tablet_manager.cc:1397] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.009s
I20260929 20:53:06.331450 2632 consensus_queue.cc:260] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:06.332244 2633 raft_consensus.cc:348] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:06.332789 2633 raft_consensus.cc:374] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:06.332809 2632 ts_tablet_manager.cc:1428] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 20:53:06.332929 2633 raft_consensus.cc:729] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ffffb3986541a7a50d7afa60d83e94, State: Initialized, Role: FOLLOWER
I20260929 20:53:06.333242 2633 consensus_queue.cc:260] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:06.334112 2681 tablet_replica.cc:330] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94: stopping tablet replica
I20260929 20:53:06.335153 2681 raft_consensus.cc:2229] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:06.335966 2633 ts_tablet_manager.cc:1428] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 20:53:06.336077 2681 raft_consensus.cc:2258] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:06.340974 2680 ts_tablet_manager.cc:1918] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:53:06.344333 2679 ts_tablet_manager.cc:1918] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:53:06.345952 2681 ts_tablet_manager.cc:1905] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:06.346331 2679 log.cc:1196] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/6673f54f330f4085958e63582aed4389
I20260929 20:53:06.346511 2680 log.cc:1196] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/6673f54f330f4085958e63582aed4389
I20260929 20:53:06.347512 2679 ts_tablet_manager.cc:1939] T 6673f54f330f4085958e63582aed4389 P a167d1b3dc124c49bf3e69dd639c2c63: Deleting consensus metadata
I20260929 20:53:06.347836 2680 ts_tablet_manager.cc:1939] T 6673f54f330f4085958e63582aed4389 P 2a5a067ca2b74b7e8af4dffbca74ab25: Deleting consensus metadata
I20260929 20:53:06.352288 2111 catalog_manager.cc:4841] TS 2a5a067ca2b74b7e8af4dffbca74ab25 (127.0.97.3:45621): tablet 6673f54f330f4085958e63582aed4389 (table test-table [id=11a6bd8abb7d4c30a7258108f8daabfc]) successfully deleted
I20260929 20:53:06.353449 2112 catalog_manager.cc:4841] TS a167d1b3dc124c49bf3e69dd639c2c63 (127.0.97.1:32949): tablet 6673f54f330f4085958e63582aed4389 (table test-table [id=11a6bd8abb7d4c30a7258108f8daabfc]) successfully deleted
I20260929 20:53:06.354357 2631 tablet_bootstrap.cc:492] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63: No bootstrap required, opened a new log
I20260929 20:53:06.354555 2631 ts_tablet_manager.cc:1397] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63: Time spent bootstrapping tablet: real 0.042s user 0.011s sys 0.015s
I20260929 20:53:06.355325 2631 raft_consensus.cc:348] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:06.355578 2631 raft_consensus.cc:374] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:06.355674 2631 raft_consensus.cc:729] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a167d1b3dc124c49bf3e69dd639c2c63, State: Initialized, Role: FOLLOWER
I20260929 20:53:06.355928 2631 consensus_queue.cc:260] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:06.356773 2631 ts_tablet_manager.cc:1428] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:53:06.358250 2681 ts_tablet_manager.cc:1918] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:53:06.358460 2681 log.cc:1196] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/6673f54f330f4085958e63582aed4389
I20260929 20:53:06.358996 2681 ts_tablet_manager.cc:1939] T 6673f54f330f4085958e63582aed4389 P a5ffffb3986541a7a50d7afa60d83e94: Deleting consensus metadata
I20260929 20:53:06.362167 2111 catalog_manager.cc:4841] TS a5ffffb3986541a7a50d7afa60d83e94 (127.0.97.2:34179): tablet 6673f54f330f4085958e63582aed4389 (table test-table [id=11a6bd8abb7d4c30a7258108f8daabfc]) successfully deleted
I20260929 20:53:06.533018 2671 raft_consensus.cc:482] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:53:06.533215 2671 raft_consensus.cc:504] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:06.533890 2671 leader_election.cc:288] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a167d1b3dc124c49bf3e69dd639c2c63 (127.0.97.1:32949), a5ffffb3986541a7a50d7afa60d83e94 (127.0.97.2:34179)
I20260929 20:53:06.534500 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171394b0f761452bb222f1240d18e8ae" candidate_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" is_pre_election: true
I20260929 20:53:06.534512 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171394b0f761452bb222f1240d18e8ae" candidate_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ffffb3986541a7a50d7afa60d83e94" is_pre_election: true
I20260929 20:53:06.534763 2422 raft_consensus.cc:2454] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a5a067ca2b74b7e8af4dffbca74ab25 in term 0.
I20260929 20:53:06.534766 2288 raft_consensus.cc:2454] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a5a067ca2b74b7e8af4dffbca74ab25 in term 0.
I20260929 20:53:06.535158 2494 leader_election.cc:302] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a5a067ca2b74b7e8af4dffbca74ab25, a167d1b3dc124c49bf3e69dd639c2c63; no voters:
I20260929 20:53:06.535418 2671 raft_consensus.cc:2781] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:53:06.535532 2671 raft_consensus.cc:482] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:53:06.535620 2671 raft_consensus.cc:3037] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:06.537727 2671 raft_consensus.cc:504] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:06.538327 2671 leader_election.cc:288] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [CANDIDATE]: Term 1 election: Requested vote from peers a167d1b3dc124c49bf3e69dd639c2c63 (127.0.97.1:32949), a5ffffb3986541a7a50d7afa60d83e94 (127.0.97.2:34179)
I20260929 20:53:06.538838 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171394b0f761452bb222f1240d18e8ae" candidate_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ffffb3986541a7a50d7afa60d83e94"
I20260929 20:53:06.538856 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171394b0f761452bb222f1240d18e8ae" candidate_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a167d1b3dc124c49bf3e69dd639c2c63"
I20260929 20:53:06.539016 2422 raft_consensus.cc:3037] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:06.539103 2288 raft_consensus.cc:3037] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:06.541486 2422 raft_consensus.cc:2454] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a5a067ca2b74b7e8af4dffbca74ab25 in term 1.
I20260929 20:53:06.541996 2493 leader_election.cc:302] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a5a067ca2b74b7e8af4dffbca74ab25, a5ffffb3986541a7a50d7afa60d83e94; no voters:
I20260929 20:53:06.542246 2288 raft_consensus.cc:2454] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a5a067ca2b74b7e8af4dffbca74ab25 in term 1.
I20260929 20:53:06.542397 2671 raft_consensus.cc:2781] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:06.542536 2671 raft_consensus.cc:686] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 LEADER]: Becoming Leader. State: Replica: 2a5a067ca2b74b7e8af4dffbca74ab25, State: Running, Role: LEADER
I20260929 20:53:06.542798 2671 consensus_queue.cc:237] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:06.545384 2126 catalog_manager.cc:5495] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a5a067ca2b74b7e8af4dffbca74ab25 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } health_report { overall_health: HEALTHY } } }
I20260929 20:53:06.622423 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:06.617429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.0.97.3:45621)
0929 20:53:06.617607 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:06.617614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:06.617698 (+ 84us) server_negotiation.cc:407] Connection header received
0929 20:53:06.617845 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:06.617851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:06.617933 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:06.618034 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:06.618357 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.619401 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.620623 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.621231 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.621357 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:06.621734 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:06.621859 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:06.621948 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:06.622022 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 20:53:06.622098 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:53:06.635192 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:06.630142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32913 (local address 127.0.97.3:45621)
0929 20:53:06.630649 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:06.630654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:06.630668 (+ 14us) server_negotiation.cc:407] Connection header received
0929 20:53:06.630715 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:06.630720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:06.630789 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:06.630876 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:06.631155 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.632168 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.633350 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.634001 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.634121 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:06.634480 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:06.634566 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:06.634648 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:06.634791 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 20:53:06.634862 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":406}
I20260929 20:53:06.635234 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:06.630122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45621 (local address 127.0.97.3:32913)
0929 20:53:06.630331 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:06.630349 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:06.630450 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:06.630898 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:06.630901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:06.630911 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:53:06.631034 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:06.631041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.632308 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.632311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:06.633243 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:06.633254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:06.634118 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:06.634122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:06.634306 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:06.634338 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:53:06.634673 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:53:06.634684 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:53:06.634776 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 20:53:06.634909 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":19}
I20260929 20:53:06.642321 2288 raft_consensus.cc:1261] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 FOLLOWER]: Refusing update from remote peer 2a5a067ca2b74b7e8af4dffbca74ab25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:06.642771 2671 consensus_queue.cc:1035] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:06.643081 2422 raft_consensus.cc:1261] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94 [term 1 FOLLOWER]: Refusing update from remote peer 2a5a067ca2b74b7e8af4dffbca74ab25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:06.643575 2654 consensus_queue.cc:1035] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:53:06.672866 2659 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 20:53:08.279879 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:08.280153 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:08.284163 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a: Sending DeleteTablet for 3 replicas of tablet 171394b0f761452bb222f1240d18e8ae
I20260929 20:53:08.284917 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet 171394b0f761452bb222f1240d18e8ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46854
I20260929 20:53:08.284916 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 171394b0f761452bb222f1240d18e8ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57422
I20260929 20:53:08.285004 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet 171394b0f761452bb222f1240d18e8ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51006
I20260929 20:53:08.302820 2711 tablet_replica.cc:330] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25: stopping tablet replica
I20260929 20:53:08.303058 2709 tablet_replica.cc:330] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94: stopping tablet replica
I20260929 20:53:08.303193 2711 raft_consensus.cc:2229] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:53:08.303287 2709 raft_consensus.cc:2229] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:08.303567 2709 raft_consensus.cc:2258] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:08.303928 2711 raft_consensus.cc:2258] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:08.304109 2710 tablet_replica.cc:330] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63: stopping tablet replica
I20260929 20:53:08.304294 2710 raft_consensus.cc:2229] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:08.304510 2710 raft_consensus.cc:2258] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:08.304949 2711 ts_tablet_manager.cc:1905] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:08.305436 2710 ts_tablet_manager.cc:1905] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:08.305433 2709 ts_tablet_manager.cc:1905] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:08.311296 2711 ts_tablet_manager.cc:1918] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:53:08.311400 2711 log.cc:1196] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/171394b0f761452bb222f1240d18e8ae
I20260929 20:53:08.311713 2711 ts_tablet_manager.cc:1939] T 171394b0f761452bb222f1240d18e8ae P 2a5a067ca2b74b7e8af4dffbca74ab25: Deleting consensus metadata
I20260929 20:53:08.311697 2710 ts_tablet_manager.cc:1918] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:53:08.311700 2709 ts_tablet_manager.cc:1918] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:53:08.311806 2710 log.cc:1196] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/171394b0f761452bb222f1240d18e8ae
I20260929 20:53:08.311808 2709 log.cc:1196] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/171394b0f761452bb222f1240d18e8ae
I20260929 20:53:08.312204 2709 ts_tablet_manager.cc:1939] T 171394b0f761452bb222f1240d18e8ae P a5ffffb3986541a7a50d7afa60d83e94: Deleting consensus metadata
I20260929 20:53:08.312304 2710 ts_tablet_manager.cc:1939] T 171394b0f761452bb222f1240d18e8ae P a167d1b3dc124c49bf3e69dd639c2c63: Deleting consensus metadata
I20260929 20:53:08.312937 2111 catalog_manager.cc:4841] TS 2a5a067ca2b74b7e8af4dffbca74ab25 (127.0.97.3:45621): tablet 171394b0f761452bb222f1240d18e8ae (table test-table [id=8a57ace708544d94aee35dd6dc341ee9]) successfully deleted
I20260929 20:53:08.313282 2112 catalog_manager.cc:4841] TS a167d1b3dc124c49bf3e69dd639c2c63 (127.0.97.1:32949): tablet 171394b0f761452bb222f1240d18e8ae (table test-table [id=8a57ace708544d94aee35dd6dc341ee9]) successfully deleted
I20260929 20:53:08.313572 2111 catalog_manager.cc:4841] TS a5ffffb3986541a7a50d7afa60d83e94 (127.0.97.2:34179): tablet 171394b0f761452bb222f1240d18e8ae (table test-table [id=8a57ace708544d94aee35dd6dc341ee9]) successfully deleted
Sep 29 20:53:08 dist-test-slave-m9g1 krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715188, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:53:08 dist-test-slave-m9g1 krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715188, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:08.342841 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:08.326165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58602 (local address 127.0.97.62:37599)
0929 20:53:08.326768 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:08.326774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:08.326862 (+ 88us) server_negotiation.cc:407] Connection header received
0929 20:53:08.327045 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:08.327050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:08.327110 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:08.327196 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:08.328472 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:08.329209 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:08.330518 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:08.330900 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:08.331004 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:08.338063 (+ 7059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:08.338068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:08.338071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:08.338105 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:08.340408 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:08.341018 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:08.341023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:08.341025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:08.341133 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:08.341457 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:08.341461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:08.341463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:08.341772 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:08.341994 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:08.342312 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 20:53:08.342513 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":263,"threads_started":1}
I20260929 20:53:08.345968 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:53:08.346540 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:53:08.352671 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 7c38f05764464eeca4f2dfd30f1f72b4 (DEFAULT_TABLE table=test-table [id=51b190f96ec94ba5ac1a14a35916404e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:08.353075 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c38f05764464eeca4f2dfd30f1f72b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:08.353266 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 7c38f05764464eeca4f2dfd30f1f72b4 (DEFAULT_TABLE table=test-table [id=51b190f96ec94ba5ac1a14a35916404e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:08.353667 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c38f05764464eeca4f2dfd30f1f72b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:08.357944 2720 tablet_bootstrap.cc:492] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63: Bootstrap starting.
I20260929 20:53:08.358175 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 7c38f05764464eeca4f2dfd30f1f72b4 (DEFAULT_TABLE table=test-table [id=51b190f96ec94ba5ac1a14a35916404e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:08.358623 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c38f05764464eeca4f2dfd30f1f72b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:08.359736 2720 tablet_bootstrap.cc:654] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:08.361619 2720 tablet_bootstrap.cc:492] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63: No bootstrap required, opened a new log
I20260929 20:53:08.361745 2720 ts_tablet_manager.cc:1397] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 20:53:08.362866 2720 raft_consensus.cc:348] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:08.363050 2720 raft_consensus.cc:374] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:08.363291 2720 raft_consensus.cc:729] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a167d1b3dc124c49bf3e69dd639c2c63, State: Initialized, Role: FOLLOWER
I20260929 20:53:08.363395 2723 tablet_bootstrap.cc:492] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94: Bootstrap starting.
I20260929 20:53:08.363482 2721 tablet_bootstrap.cc:492] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25: Bootstrap starting.
I20260929 20:53:08.363572 2720 consensus_queue.cc:260] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:08.364795 2720 ts_tablet_manager.cc:1428] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 20:53:08.364903 2723 tablet_bootstrap.cc:654] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:08.364933 2721 tablet_bootstrap.cc:654] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:08.367345 2721 tablet_bootstrap.cc:492] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25: No bootstrap required, opened a new log
I20260929 20:53:08.367336 2723 tablet_bootstrap.cc:492] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94: No bootstrap required, opened a new log
I20260929 20:53:08.367467 2723 ts_tablet_manager.cc:1397] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 20:53:08.367467 2721 ts_tablet_manager.cc:1397] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260929 20:53:08.368264 2721 raft_consensus.cc:348] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:08.368484 2721 raft_consensus.cc:374] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:08.368611 2721 raft_consensus.cc:729] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a5a067ca2b74b7e8af4dffbca74ab25, State: Initialized, Role: FOLLOWER
I20260929 20:53:08.368673 2723 raft_consensus.cc:348] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:08.368880 2723 raft_consensus.cc:374] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:08.368963 2723 raft_consensus.cc:729] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ffffb3986541a7a50d7afa60d83e94, State: Initialized, Role: FOLLOWER
I20260929 20:53:08.368846 2721 consensus_queue.cc:260] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:08.369362 2721 ts_tablet_manager.cc:1428] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 20:53:08.369575 2723 consensus_queue.cc:260] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:08.370309 2723 ts_tablet_manager.cc:1428] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 20:53:08.391027 2724 raft_consensus.cc:482] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:53:08.391196 2724 raft_consensus.cc:504] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:08.391770 2724 leader_election.cc:288] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5ffffb3986541a7a50d7afa60d83e94 (127.0.97.2:34179), 2a5a067ca2b74b7e8af4dffbca74ab25 (127.0.97.3:45621)
I20260929 20:53:08.392302 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c38f05764464eeca4f2dfd30f1f72b4" candidate_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" is_pre_election: true
I20260929 20:53:08.392324 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c38f05764464eeca4f2dfd30f1f72b4" candidate_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ffffb3986541a7a50d7afa60d83e94" is_pre_election: true
I20260929 20:53:08.392541 2422 raft_consensus.cc:2454] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a167d1b3dc124c49bf3e69dd639c2c63 in term 0.
I20260929 20:53:08.392541 2563 raft_consensus.cc:2454] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a167d1b3dc124c49bf3e69dd639c2c63 in term 0.
I20260929 20:53:08.392995 2214 leader_election.cc:302] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a167d1b3dc124c49bf3e69dd639c2c63, a5ffffb3986541a7a50d7afa60d83e94; no voters:
I20260929 20:53:08.393263 2724 raft_consensus.cc:2781] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:53:08.393362 2724 raft_consensus.cc:482] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:53:08.393443 2724 raft_consensus.cc:3037] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:08.395422 2724 raft_consensus.cc:504] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:08.395910 2724 leader_election.cc:288] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [CANDIDATE]: Term 1 election: Requested vote from peers a5ffffb3986541a7a50d7afa60d83e94 (127.0.97.2:34179), 2a5a067ca2b74b7e8af4dffbca74ab25 (127.0.97.3:45621)
I20260929 20:53:08.396349 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c38f05764464eeca4f2dfd30f1f72b4" candidate_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ffffb3986541a7a50d7afa60d83e94"
I20260929 20:53:08.396420 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c38f05764464eeca4f2dfd30f1f72b4" candidate_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25"
I20260929 20:53:08.396500 2422 raft_consensus.cc:3037] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:08.396603 2563 raft_consensus.cc:3037] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:08.398425 2422 raft_consensus.cc:2454] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a167d1b3dc124c49bf3e69dd639c2c63 in term 1.
I20260929 20:53:08.398628 2563 raft_consensus.cc:2454] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a167d1b3dc124c49bf3e69dd639c2c63 in term 1.
I20260929 20:53:08.398820 2214 leader_election.cc:302] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a167d1b3dc124c49bf3e69dd639c2c63, a5ffffb3986541a7a50d7afa60d83e94; no voters:
I20260929 20:53:08.399109 2724 raft_consensus.cc:2781] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:08.399505 2724 raft_consensus.cc:686] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 LEADER]: Becoming Leader. State: Replica: a167d1b3dc124c49bf3e69dd639c2c63, State: Running, Role: LEADER
I20260929 20:53:08.399775 2724 consensus_queue.cc:237] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:08.402453 2127 catalog_manager.cc:5495] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 reported cstate change: term changed from 0 to 1, leader changed from <none> to a167d1b3dc124c49bf3e69dd639c2c63 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } health_report { overall_health: UNKNOWN } } }
I20260929 20:53:08.431878 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:08.427246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57450 (local address 127.0.97.1:32949)
0929 20:53:08.427800 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:08.427804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:08.427821 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:53:08.427876 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:08.427879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:08.427939 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:08.428012 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:08.428300 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:08.429188 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:08.430531 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:08.430909 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:08.430993 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:08.431325 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:08.431414 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:08.431472 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:08.431571 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 20:53:08.431626 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":335,"threads_started":1}
I20260929 20:53:08.440080 2563 raft_consensus.cc:1261] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 FOLLOWER]: Refusing update from remote peer a167d1b3dc124c49bf3e69dd639c2c63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:08.440099 2422 raft_consensus.cc:1261] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94 [term 1 FOLLOWER]: Refusing update from remote peer a167d1b3dc124c49bf3e69dd639c2c63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:08.440577 2728 consensus_queue.cc:1035] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:08.440829 2724 consensus_queue.cc:1035] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:08.446063 2694 mvcc.cc:205] Tried to move back new op lower bound from 7334769411846733824 to 7334769411687763968. Current Snapshot: MvccSnapshot[applied={T|T < 7334769411846733824}]
I20260929 20:53:08.483973 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:08.484266 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:08.487980 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a: Sending DeleteTablet for 3 replicas of tablet 7c38f05764464eeca4f2dfd30f1f72b4
I20260929 20:53:08.488730 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c38f05764464eeca4f2dfd30f1f72b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46854
I20260929 20:53:08.488801 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c38f05764464eeca4f2dfd30f1f72b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57422
I20260929 20:53:08.489009 2709 tablet_replica.cc:330] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94: stopping tablet replica
I20260929 20:53:08.489058 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c38f05764464eeca4f2dfd30f1f72b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51006
I20260929 20:53:08.489189 2709 raft_consensus.cc:2229] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:08.489362 2709 raft_consensus.cc:2258] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:08.489594 2711 tablet_replica.cc:330] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25: stopping tablet replica
I20260929 20:53:08.489768 2711 raft_consensus.cc:2229] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:08.489938 2711 raft_consensus.cc:2258] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:08.491020 2709 ts_tablet_manager.cc:1905] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:08.491513 2710 tablet_replica.cc:330] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63: stopping tablet replica
I20260929 20:53:08.491683 2710 raft_consensus.cc:2229] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:53:08.491969 2710 raft_consensus.cc:2258] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:08.492724 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 20:53:08.492920 2710 ts_tablet_manager.cc:1905] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 20:53:08.493651 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:53:08.496881 2709 ts_tablet_manager.cc:1918] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:53:08.497017 2709 log.cc:1196] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/7c38f05764464eeca4f2dfd30f1f72b4
I20260929 20:53:08.497362 2709 ts_tablet_manager.cc:1939] T 7c38f05764464eeca4f2dfd30f1f72b4 P a5ffffb3986541a7a50d7afa60d83e94: Deleting consensus metadata
I20260929 20:53:08.498469 2111 catalog_manager.cc:4841] TS a5ffffb3986541a7a50d7afa60d83e94 (127.0.97.2:34179): tablet 7c38f05764464eeca4f2dfd30f1f72b4 (table test-table [id=51b190f96ec94ba5ac1a14a35916404e]) successfully deleted
I20260929 20:53:08.498898 2711 ts_tablet_manager.cc:1905] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:08.498951 2710 ts_tablet_manager.cc:1918] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:53:08.499073 2710 log.cc:1196] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/7c38f05764464eeca4f2dfd30f1f72b4
I20260929 20:53:08.499440 2710 ts_tablet_manager.cc:1939] T 7c38f05764464eeca4f2dfd30f1f72b4 P a167d1b3dc124c49bf3e69dd639c2c63: Deleting consensus metadata
I20260929 20:53:08.500488 2112 catalog_manager.cc:4841] TS a167d1b3dc124c49bf3e69dd639c2c63 (127.0.97.1:32949): tablet 7c38f05764464eeca4f2dfd30f1f72b4 (table test-table [id=51b190f96ec94ba5ac1a14a35916404e]) successfully deleted
I20260929 20:53:08.501883 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 7ccd3fa8f46146baa0ebde017c0d8f90 (DEFAULT_TABLE table=test-table [id=af79267f679745579db32ef542ca14c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:08.502338 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ccd3fa8f46146baa0ebde017c0d8f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:08.503940 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 7ccd3fa8f46146baa0ebde017c0d8f90 (DEFAULT_TABLE table=test-table [id=af79267f679745579db32ef542ca14c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:08.504311 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ccd3fa8f46146baa0ebde017c0d8f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:08.504974 2711 ts_tablet_manager.cc:1918] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:53:08.505119 2711 log.cc:1196] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/7c38f05764464eeca4f2dfd30f1f72b4
I20260929 20:53:08.505532 2711 ts_tablet_manager.cc:1939] T 7c38f05764464eeca4f2dfd30f1f72b4 P 2a5a067ca2b74b7e8af4dffbca74ab25: Deleting consensus metadata
I20260929 20:53:08.506474 2111 catalog_manager.cc:4841] TS 2a5a067ca2b74b7e8af4dffbca74ab25 (127.0.97.3:45621): tablet 7c38f05764464eeca4f2dfd30f1f72b4 (table test-table [id=51b190f96ec94ba5ac1a14a35916404e]) successfully deleted
I20260929 20:53:08.507181 2723 tablet_bootstrap.cc:492] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94: Bootstrap starting.
I20260929 20:53:08.508684 2723 tablet_bootstrap.cc:654] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:08.510483 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 7ccd3fa8f46146baa0ebde017c0d8f90 (DEFAULT_TABLE table=test-table [id=af79267f679745579db32ef542ca14c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:08.510847 2723 tablet_bootstrap.cc:492] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94: No bootstrap required, opened a new log
I20260929 20:53:08.510874 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ccd3fa8f46146baa0ebde017c0d8f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:08.510993 2723 ts_tablet_manager.cc:1397] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 20:53:08.511639 2723 raft_consensus.cc:348] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:08.511878 2723 raft_consensus.cc:374] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:08.511951 2723 raft_consensus.cc:729] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ffffb3986541a7a50d7afa60d83e94, State: Initialized, Role: FOLLOWER
I20260929 20:53:08.512192 2723 consensus_queue.cc:260] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:08.512673 2723 ts_tablet_manager.cc:1428] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:53:08.514503 2721 tablet_bootstrap.cc:492] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25: Bootstrap starting.
I20260929 20:53:08.515758 2721 tablet_bootstrap.cc:654] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:08.519090 2721 tablet_bootstrap.cc:492] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25: No bootstrap required, opened a new log
I20260929 20:53:08.519233 2721 ts_tablet_manager.cc:1397] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260929 20:53:08.519807 2720 tablet_bootstrap.cc:492] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63: Bootstrap starting.
I20260929 20:53:08.519918 2721 raft_consensus.cc:348] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:08.520147 2721 raft_consensus.cc:374] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:08.520226 2721 raft_consensus.cc:729] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a5a067ca2b74b7e8af4dffbca74ab25, State: Initialized, Role: FOLLOWER
I20260929 20:53:08.520443 2721 consensus_queue.cc:260] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:08.520808 2721 ts_tablet_manager.cc:1428] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 20:53:08.521330 2720 tablet_bootstrap.cc:654] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:08.523805 2720 tablet_bootstrap.cc:492] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63: No bootstrap required, opened a new log
I20260929 20:53:08.523962 2720 ts_tablet_manager.cc:1397] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260929 20:53:08.524646 2720 raft_consensus.cc:348] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:08.524821 2720 raft_consensus.cc:374] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:08.524887 2720 raft_consensus.cc:729] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a167d1b3dc124c49bf3e69dd639c2c63, State: Initialized, Role: FOLLOWER
I20260929 20:53:08.525095 2720 consensus_queue.cc:260] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:08.526562 2720 ts_tablet_manager.cc:1428] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 20:53:08.682101 2704 raft_consensus.cc:482] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:53:08.682331 2704 raft_consensus.cc:504] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:08.683043 2704 leader_election.cc:288] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a167d1b3dc124c49bf3e69dd639c2c63 (127.0.97.1:32949), a5ffffb3986541a7a50d7afa60d83e94 (127.0.97.2:34179)
I20260929 20:53:08.683624 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ccd3fa8f46146baa0ebde017c0d8f90" candidate_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" is_pre_election: true
I20260929 20:53:08.683640 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ccd3fa8f46146baa0ebde017c0d8f90" candidate_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ffffb3986541a7a50d7afa60d83e94" is_pre_election: true
I20260929 20:53:08.683851 2288 raft_consensus.cc:2454] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a5a067ca2b74b7e8af4dffbca74ab25 in term 0.
I20260929 20:53:08.683858 2422 raft_consensus.cc:2454] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a5a067ca2b74b7e8af4dffbca74ab25 in term 0.
I20260929 20:53:08.684356 2494 leader_election.cc:302] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a5a067ca2b74b7e8af4dffbca74ab25, a167d1b3dc124c49bf3e69dd639c2c63; no voters:
I20260929 20:53:08.684625 2704 raft_consensus.cc:2781] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:53:08.684738 2704 raft_consensus.cc:482] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:53:08.684816 2704 raft_consensus.cc:3037] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:08.687691 2704 raft_consensus.cc:504] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:08.688323 2704 leader_election.cc:288] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [CANDIDATE]: Term 1 election: Requested vote from peers a167d1b3dc124c49bf3e69dd639c2c63 (127.0.97.1:32949), a5ffffb3986541a7a50d7afa60d83e94 (127.0.97.2:34179)
I20260929 20:53:08.688795 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ccd3fa8f46146baa0ebde017c0d8f90" candidate_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a167d1b3dc124c49bf3e69dd639c2c63"
I20260929 20:53:08.688822 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ccd3fa8f46146baa0ebde017c0d8f90" candidate_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ffffb3986541a7a50d7afa60d83e94"
I20260929 20:53:08.688993 2288 raft_consensus.cc:3037] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:08.688992 2422 raft_consensus.cc:3037] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:08.690907 2422 raft_consensus.cc:2454] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a5a067ca2b74b7e8af4dffbca74ab25 in term 1.
I20260929 20:53:08.691286 2493 leader_election.cc:302] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a5a067ca2b74b7e8af4dffbca74ab25, a5ffffb3986541a7a50d7afa60d83e94; no voters:
I20260929 20:53:08.691849 2288 raft_consensus.cc:2454] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a5a067ca2b74b7e8af4dffbca74ab25 in term 1.
I20260929 20:53:08.692129 2704 raft_consensus.cc:2781] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:08.692296 2704 raft_consensus.cc:686] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 LEADER]: Becoming Leader. State: Replica: 2a5a067ca2b74b7e8af4dffbca74ab25, State: Running, Role: LEADER
I20260929 20:53:08.692552 2704 consensus_queue.cc:237] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } }
I20260929 20:53:08.695220 2127 catalog_manager.cc:5495] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a5a067ca2b74b7e8af4dffbca74ab25 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a5a067ca2b74b7e8af4dffbca74ab25" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45621 } health_report { overall_health: HEALTHY } } }
I20260929 20:53:08.759567 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:08.754455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51034 (local address 127.0.97.3:45621)
0929 20:53:08.755400 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:08.755404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:08.755420 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:53:08.755464 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:08.755467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:08.755528 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:08.755610 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:08.755975 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:08.756705 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:08.757957 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:08.758392 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:08.758495 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:08.758898 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:08.758991 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:08.759061 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:08.759220 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 20:53:08.759296 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":736,"threads_started":1}
I20260929 20:53:08.767045 2288 raft_consensus.cc:1261] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 FOLLOWER]: Refusing update from remote peer 2a5a067ca2b74b7e8af4dffbca74ab25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:08.767045 2422 raft_consensus.cc:1261] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94 [term 1 FOLLOWER]: Refusing update from remote peer 2a5a067ca2b74b7e8af4dffbca74ab25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:08.767478 2704 consensus_queue.cc:1035] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a167d1b3dc124c49bf3e69dd639c2c63" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:08.767731 2654 consensus_queue.cc:1035] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ffffb3986541a7a50d7afa60d83e94" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:53:08.785106 2690 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 20:53:10.355508 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:10.355705 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:10.359310 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9de46b11bfeb4eeba2a2093b3e1f713a: Sending DeleteTablet for 3 replicas of tablet 7ccd3fa8f46146baa0ebde017c0d8f90
I20260929 20:53:10.360069 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ccd3fa8f46146baa0ebde017c0d8f90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57422
I20260929 20:53:10.360139 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ccd3fa8f46146baa0ebde017c0d8f90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46854
I20260929 20:53:10.360638 2764 tablet_replica.cc:330] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63: stopping tablet replica
I20260929 20:53:10.360627 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ccd3fa8f46146baa0ebde017c0d8f90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51006
I20260929 20:53:10.360733 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 2198
I20260929 20:53:10.360951 2764 raft_consensus.cc:2229] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:10.361800 2764 raft_consensus.cc:2258] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:10.363165 2764 ts_tablet_manager.cc:1905] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:10.370332 2764 ts_tablet_manager.cc:1918] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:53:10.370550 2764 log.cc:1196] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/7ccd3fa8f46146baa0ebde017c0d8f90
I20260929 20:53:10.371068 2764 ts_tablet_manager.cc:1939] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a167d1b3dc124c49bf3e69dd639c2c63: Deleting consensus metadata
I20260929 20:53:10.371613 2765 tablet_replica.cc:330] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94: stopping tablet replica
I20260929 20:53:10.371810 2765 raft_consensus.cc:2229] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:10.372056 2765 raft_consensus.cc:2258] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:10.372378 2112 catalog_manager.cc:4841] TS a167d1b3dc124c49bf3e69dd639c2c63 (127.0.97.1:32949): tablet 7ccd3fa8f46146baa0ebde017c0d8f90 (table test-table [id=af79267f679745579db32ef542ca14c4]) successfully deleted
I20260929 20:53:10.372649 2766 tablet_replica.cc:330] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25: stopping tablet replica
I20260929 20:53:10.372825 2766 raft_consensus.cc:2229] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:53:10.373168 2766 raft_consensus.cc:2258] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:10.373212 2765 ts_tablet_manager.cc:1905] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:10.374068 2766 ts_tablet_manager.cc:1905] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:10.378300 2765 ts_tablet_manager.cc:1918] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:53:10.378414 2765 log.cc:1196] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/7ccd3fa8f46146baa0ebde017c0d8f90
I20260929 20:53:10.378718 2766 ts_tablet_manager.cc:1918] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:53:10.378756 2765 ts_tablet_manager.cc:1939] T 7ccd3fa8f46146baa0ebde017c0d8f90 P a5ffffb3986541a7a50d7afa60d83e94: Deleting consensus metadata
I20260929 20:53:10.378810 2766 log.cc:1196] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/7ccd3fa8f46146baa0ebde017c0d8f90
I20260929 20:53:10.379142 2766 ts_tablet_manager.cc:1939] T 7ccd3fa8f46146baa0ebde017c0d8f90 P 2a5a067ca2b74b7e8af4dffbca74ab25: Deleting consensus metadata
I20260929 20:53:10.379667 2111 catalog_manager.cc:4841] TS a5ffffb3986541a7a50d7afa60d83e94 (127.0.97.2:34179): tablet 7ccd3fa8f46146baa0ebde017c0d8f90 (table test-table [id=af79267f679745579db32ef542ca14c4]) successfully deleted
I20260929 20:53:10.379997 2111 catalog_manager.cc:4841] TS 2a5a067ca2b74b7e8af4dffbca74ab25 (127.0.97.3:45621): tablet 7ccd3fa8f46146baa0ebde017c0d8f90 (table test-table [id=af79267f679745579db32ef542ca14c4]) successfully deleted
Sep 29 20:53:10 dist-test-slave-m9g1 krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715188, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:53:10.391274 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:10.363745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53182 (local address 127.0.97.1:32949)
0929 20:53:10.371426 (+ 7681us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:10.371431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:10.371450 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:53:10.371514 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:10.371519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:10.371587 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:10.371690 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:10.373368 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:10.374412 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:10.376137 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:10.376683 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:10.376802 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:10.386241 (+ 9439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:10.386256 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:10.386273 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:10.386317 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:10.388941 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:10.389506 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:10.389513 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:10.389515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:10.389633 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:10.389909 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:10.389912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:10.389913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:10.390279 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:10.390486 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:10.390828 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 20:53:10.391000 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7561,"thread_start_us":7432,"threads_started":1}
W20260929 20:53:10.391870 2329 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53182
W20260929 20:53:10.392428 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 2198: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:53:10.392614 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 2198
I20260929 20:53:10.409927 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 2338
Sep 29 20:53:10 dist-test-slave-m9g1 krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715188, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:53:10.423658 2769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:10.410687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58802 (local address 127.0.97.2:34179)
0929 20:53:10.411168 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:10.411172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:10.411187 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:53:10.411237 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:10.411240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:10.411296 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:10.411404 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:10.412568 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:10.413286 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:10.414567 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:10.414929 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:10.415048 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:10.418791 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:10.418806 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:10.418821 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:10.418855 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:10.421294 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:10.421866 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:10.421873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:10.421875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:10.422025 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:10.422405 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:10.422408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:10.422410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:10.422720 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:10.422940 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:10.423183 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 20:53:10.423340 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":238,"threads_started":1}
W20260929 20:53:10.424297 2468 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58802
W20260929 20:53:10.424858 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 2338: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:53:10.425007 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 2338
I20260929 20:53:10.438865 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 2477
Sep 29 20:53:10 dist-test-slave-m9g1 krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715188, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:53:10.452275 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:10.439636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51310 (local address 127.0.97.3:45621)
0929 20:53:10.440106 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:10.440110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:10.440124 (+ 14us) server_negotiation.cc:407] Connection header received
0929 20:53:10.440175 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:10.440179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:10.440234 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:10.440319 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:10.441441 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:10.442178 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:10.443447 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:10.443852 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:10.443948 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:10.447642 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:10.447656 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:10.447671 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:10.447707 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:10.450271 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:10.450788 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:10.450794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:10.450796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:10.450905 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:10.451157 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:10.451159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:10.451161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:10.451482 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:10.451684 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:10.451885 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 20:53:10.452029 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":249,"threads_started":1}
W20260929 20:53:10.452860 2607 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51310
W20260929 20:53:10.453399 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 2477: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:53:10.453532 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 2477
I20260929 20:53:10.473052 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 2095
I20260929 20:53:10.474331 2167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:10.588238 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 2095
2026-09-29T20:53:10Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8073 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:53:10 dist-test-slave-m9g1 krb5kdc[2774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:53:10 dist-test-slave-m9g1 krb5kdc[2774](info): set up 2 sockets
Sep 29 20:53:10 dist-test-slave-m9g1 krb5kdc[2774](info): commencing operation
krb5kdc: starting...
W20260929 20:53:11.570034 388 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:53:11 dist-test-slave-m9g1 krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:53:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:53:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:53:11.739315 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42715
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:12.039009 2787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:12.039273 2787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:12.039392 2787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:12.046383 2787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:53:12.046484 2787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:12.046550 2787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:12.046648 2787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:53:12.046702 2787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:53:12.056291 2787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:12.057554 2787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:12.058892 2787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:12.070286 2792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:12.070446 2795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:12.070261 2793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:12.071045 2787 server_base.cc:1034] running on GCE node
I20260929 20:53:12.071744 2787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:12.073129 2787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:12.074276 2787 hybrid_clock.cc:648] HybridClock initialized: now 1790715192074227 us; error 43 us; skew 500 ppm
I20260929 20:53:12.074616 2787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:12 dist-test-slave-m9g1 krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:12.077523 2787 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:53:12.079056 2787 webserver.cc:460] Webserver started at http://127.0.97.62:36327/ using document root <none> and password file <none>
I20260929 20:53:12.079782 2787 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:12.079905 2787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:12.080207 2787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:12.083187 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "071b4bd500cd4e34a2a18f56c71d55fe"
format_stamp: "Formatted at 2026-09-29 20:53:12 on dist-test-slave-m9g1"
server_key: "840586ad6d97d9dbbb985d4da0429d50"
server_key_iv: "c20ba3bf746a98b69affb738e9e1504a"
server_key_version: "encryptionkey@0"
I20260929 20:53:12.083986 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "071b4bd500cd4e34a2a18f56c71d55fe"
format_stamp: "Formatted at 2026-09-29 20:53:12 on dist-test-slave-m9g1"
server_key: "840586ad6d97d9dbbb985d4da0429d50"
server_key_iv: "c20ba3bf746a98b69affb738e9e1504a"
server_key_version: "encryptionkey@0"
I20260929 20:53:12.088971 2787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:53:12.092311 2802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:12.093624 2787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:53:12.093778 2787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "071b4bd500cd4e34a2a18f56c71d55fe"
format_stamp: "Formatted at 2026-09-29 20:53:12 on dist-test-slave-m9g1"
server_key: "840586ad6d97d9dbbb985d4da0429d50"
server_key_iv: "c20ba3bf746a98b69affb738e9e1504a"
server_key_version: "encryptionkey@0"
I20260929 20:53:12.093945 2787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:12.117170 2787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:12.120860 2787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:12.121099 2787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:12.136862 2787 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42715
I20260929 20:53:12.136904 2853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42715 every 8 connection(s)
I20260929 20:53:12.138118 2787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:53:12.141564 2854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:12.145013 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 2787
I20260929 20:53:12.145233 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:53:12.145998 388 external_mini_cluster.cc:1428] Setting key ae2fac8747bdf3f191b277678a68b77a
I20260929 20:53:12.153424 2854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe: Bootstrap starting.
I20260929 20:53:12.156343 2854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:12.157285 2854 log.cc:824] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:53:12 dist-test-slave-m9g1 krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:12.160112 2854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe: No bootstrap required, opened a new log
I20260929 20:53:12.166435 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:12.148036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.0.97.62:42715)
0929 20:53:12.148714 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:12.148725 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:12.148771 (+ 46us) server_negotiation.cc:407] Connection header received
0929 20:53:12.150313 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:12.150334 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:12.150726 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:12.151077 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:12.152462 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:12.153753 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:12.155021 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:12.155656 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:12.155818 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:12.159738 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:12.159753 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:12.159766 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:12.159800 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:12.162674 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:12.163447 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:12.163453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:12.163459 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:12.163612 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:12.163971 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:12.163973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:12.163975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:12.164469 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:12.164680 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:12.165066 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 20:53:12.165395 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":305,"threads_started":1}
I20260929 20:53:12.168403 2854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071b4bd500cd4e34a2a18f56c71d55fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42715 } }
I20260929 20:53:12.168813 2854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:12.168913 2854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 071b4bd500cd4e34a2a18f56c71d55fe, State: Initialized, Role: FOLLOWER
I20260929 20:53:12.169456 2854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071b4bd500cd4e34a2a18f56c71d55fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42715 } }
I20260929 20:53:12.169649 2854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:53:12.169739 2854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:53:12.169857 2854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:12.171820 2854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071b4bd500cd4e34a2a18f56c71d55fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42715 } }
I20260929 20:53:12.172322 2854 leader_election.cc:302] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 071b4bd500cd4e34a2a18f56c71d55fe; no voters:
I20260929 20:53:12.172845 2854 leader_election.cc:288] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:53:12.172948 2859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:12.173518 2859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe [term 1 LEADER]: Becoming Leader. State: Replica: 071b4bd500cd4e34a2a18f56c71d55fe, State: Running, Role: LEADER
I20260929 20:53:12.174048 2859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071b4bd500cd4e34a2a18f56c71d55fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42715 } }
I20260929 20:53:12.174630 2854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:53:12.176761 2861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 071b4bd500cd4e34a2a18f56c71d55fe. Latest consensus state: current_term: 1 leader_uuid: "071b4bd500cd4e34a2a18f56c71d55fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071b4bd500cd4e34a2a18f56c71d55fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42715 } } }
I20260929 20:53:12.177119 2861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:12.177742 2860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "071b4bd500cd4e34a2a18f56c71d55fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071b4bd500cd4e34a2a18f56c71d55fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42715 } } }
I20260929 20:53:12.178028 2860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:12.179081 2868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:53:12.183600 2868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:53:12.193048 2868 catalog_manager.cc:1347] Generated new cluster ID: 45b0f5ebd8cc485bb2a49daf968d7373
I20260929 20:53:12.193163 2868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:53:12.208424 2868 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:53:12.209734 2868 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:53:12.220090 2868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 071b4bd500cd4e34a2a18f56c71d55fe: Generated new TSK 0
I20260929 20:53:12.220969 2868 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:53:12.261236 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42715
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:12.569167 2880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:12.569442 2880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:12.569542 2880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:12.578835 2880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:12.578935 2880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:12.579087 2880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:53:12.588714 2880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:12.589908 2880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:12.591180 2880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:12.603876 2885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:12.603911 2886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:12.603948 2888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:12.604710 2880 server_base.cc:1034] running on GCE node
I20260929 20:53:12.605316 2880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:12.606242 2880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:12.607432 2880 hybrid_clock.cc:648] HybridClock initialized: now 1790715192607393 us; error 42 us; skew 500 ppm
I20260929 20:53:12.607714 2880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:12 dist-test-slave-m9g1 krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:12.610606 2880 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:53:12.611936 2880 webserver.cc:460] Webserver started at http://127.0.97.1:41931/ using document root <none> and password file <none>
I20260929 20:53:12.612572 2880 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:12.612684 2880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:12.613036 2880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:12.615978 2880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "5d56138dbde24f0c98df185d1c281121"
format_stamp: "Formatted at 2026-09-29 20:53:12 on dist-test-slave-m9g1"
server_key: "ca7eab4fd6f222c05ed4482251111600"
server_key_iv: "5a02829e364d7f68e92aaa95d44534b3"
server_key_version: "encryptionkey@0"
I20260929 20:53:12.616775 2880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5d56138dbde24f0c98df185d1c281121"
format_stamp: "Formatted at 2026-09-29 20:53:12 on dist-test-slave-m9g1"
server_key: "ca7eab4fd6f222c05ed4482251111600"
server_key_iv: "5a02829e364d7f68e92aaa95d44534b3"
server_key_version: "encryptionkey@0"
I20260929 20:53:12.621541 2880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260929 20:53:12.624799 2895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:12.626063 2880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:53:12.626226 2880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "5d56138dbde24f0c98df185d1c281121"
format_stamp: "Formatted at 2026-09-29 20:53:12 on dist-test-slave-m9g1"
server_key: "ca7eab4fd6f222c05ed4482251111600"
server_key_iv: "5a02829e364d7f68e92aaa95d44534b3"
server_key_version: "encryptionkey@0"
I20260929 20:53:12.626426 2880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:12.639351 2880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:12.643179 2880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:12.643438 2880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:12.644261 2880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:12.645474 2880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:12.645547 2880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:12.645659 2880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:12.645720 2880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:12.674811 2880 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42201
I20260929 20:53:12.674858 3007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42201 every 8 connection(s)
I20260929 20:53:12.676131 2880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:53:12.686177 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 2880
I20260929 20:53:12.686404 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:53:12.687148 388 external_mini_cluster.cc:1428] Setting key e0548165fcd808ea74fe62087b3b3c2a
Sep 29 20:53:12 dist-test-slave-m9g1 krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:12.694411 3010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:12.678522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54739 (local address 127.0.97.62:42715)
0929 20:53:12.678925 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:12.678931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:12.679947 (+ 1016us) server_negotiation.cc:407] Connection header received
0929 20:53:12.681608 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:12.681612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:12.681673 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:12.681762 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:12.683164 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:12.683846 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:12.685236 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:12.685610 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:12.685690 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:12.689302 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:12.689307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:12.689310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:12.689339 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:12.691720 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:12.692350 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:12.692355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:12.692358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:12.692464 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:12.692840 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:12.692845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:12.692849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:12.693101 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:12.693226 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:12.693841 (+ 615us) server_negotiation.cc:299] Negotiation successful
0929 20:53:12.694039 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":194,"threads_started":1}
I20260929 20:53:12.695315 3011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:12.679107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42715 (local address 127.0.97.1:54739)
0929 20:53:12.679797 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:12.679832 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:12.681370 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:12.681947 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:12.681955 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:12.682503 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:12.683033 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:12.683049 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:12.683958 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:12.683962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:12.685135 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:12.685145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:12.685761 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:12.685765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:12.685946 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:12.686648 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:12.686676 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:12.689114 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:12.691846 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:12.691855 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:12.691870 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:12.692246 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:12.692565 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:12.692570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:12.692574 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:12.692745 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:12.693211 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:12.693218 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:12.693521 (+ 303us) client_negotiation.cc:769] Sending connection context
0929 20:53:12.693752 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 20:53:12.694251 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":250,"threads_started":1}
I20260929 20:53:12.696729 3008 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42715
I20260929 20:53:12.697036 3008 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:12.697834 3008 heartbeater.cc:511] Master 127.0.97.62:42715 requested a full tablet report, sending...
I20260929 20:53:12.700181 2819 ts_manager.cc:194] Registered new tserver with Master: 5d56138dbde24f0c98df185d1c281121 (127.0.97.1:42201)
I20260929 20:53:12.702241 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54739
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:53:12.724808 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42715
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:13.024252 3014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:13.024516 3014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:13.024614 3014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:13.031553 3014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:13.031648 3014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:13.031764 3014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:53:13.040887 3014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:13.042017 3014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:13.043213 3014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:13.054886 3020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:13.054920 3022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:13.055135 3019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:13.055757 3014 server_base.cc:1034] running on GCE node
I20260929 20:53:13.056319 3014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:13.057214 3014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:13.058391 3014 hybrid_clock.cc:648] HybridClock initialized: now 1790715193058347 us; error 37 us; skew 500 ppm
I20260929 20:53:13.058686 3014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:13 dist-test-slave-m9g1 krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:13.061528 3014 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:53:13.062824 3014 webserver.cc:460] Webserver started at http://127.0.97.2:42297/ using document root <none> and password file <none>
I20260929 20:53:13.063437 3014 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:13.063530 3014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:13.063771 3014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:13.066578 3014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "8499f1e527244cb7a90083d9b5624505"
format_stamp: "Formatted at 2026-09-29 20:53:13 on dist-test-slave-m9g1"
server_key: "e965590588b3b9eb6035b4428e247ff5"
server_key_iv: "1bd34525c74336eed23c0d7e165cd99d"
server_key_version: "encryptionkey@0"
I20260929 20:53:13.067284 3014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8499f1e527244cb7a90083d9b5624505"
format_stamp: "Formatted at 2026-09-29 20:53:13 on dist-test-slave-m9g1"
server_key: "e965590588b3b9eb6035b4428e247ff5"
server_key_iv: "1bd34525c74336eed23c0d7e165cd99d"
server_key_version: "encryptionkey@0"
I20260929 20:53:13.072080 3014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:53:13.075264 3029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:13.076563 3014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:53:13.076699 3014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "8499f1e527244cb7a90083d9b5624505"
format_stamp: "Formatted at 2026-09-29 20:53:13 on dist-test-slave-m9g1"
server_key: "e965590588b3b9eb6035b4428e247ff5"
server_key_iv: "1bd34525c74336eed23c0d7e165cd99d"
server_key_version: "encryptionkey@0"
I20260929 20:53:13.076874 3014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:13.094307 3014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:13.098062 3014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:13.098398 3014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:13.099288 3014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:13.100502 3014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:13.100567 3014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:13.100653 3014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:13.100693 3014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:13.128960 3014 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41849
I20260929 20:53:13.128983 3141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41849 every 8 connection(s)
I20260929 20:53:13.130254 3014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:53:13.137916 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 3014
I20260929 20:53:13.138101 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:53:13.138625 388 external_mini_cluster.cc:1428] Setting key c34f732fa29993c14a1f9e68a40e55df
Sep 29 20:53:13 dist-test-slave-m9g1 krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:13.150439 3010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:13.132546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50253 (local address 127.0.97.62:42715)
0929 20:53:13.132709 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:13.132714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:13.133824 (+ 1110us) server_negotiation.cc:407] Connection header received
0929 20:53:13.135609 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:13.135615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:13.135695 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:13.135780 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:13.137250 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:13.137990 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:13.139681 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:13.140343 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:13.140461 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:13.144458 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:13.144465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:13.144470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:13.144504 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:13.147186 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:13.148011 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:13.148017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:13.148021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:13.148184 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:13.148638 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:13.148643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:13.148646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:13.148966 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:13.149093 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:13.149874 (+ 781us) server_negotiation.cc:299] Negotiation successful
0929 20:53:13.150101 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:53:13.151599 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:13.133019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42715 (local address 127.0.97.2:50253)
0929 20:53:13.133676 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:13.133717 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:13.135351 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:13.136008 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:13.136016 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:13.136566 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:13.137093 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:13.137110 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:13.138207 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:13.138212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:13.139567 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:13.139581 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:13.140549 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:13.140557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:13.140822 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:13.141528 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:13.141561 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:13.144244 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:13.147313 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:13.147322 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:13.147337 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:13.147867 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:13.148292 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:13.148297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:13.148300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:13.148521 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:13.149085 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:13.149093 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:13.149476 (+ 383us) client_negotiation.cc:769] Sending connection context
0929 20:53:13.149757 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 20:53:13.150290 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":248,"threads_started":1}
I20260929 20:53:13.153090 3142 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42715
I20260929 20:53:13.153414 3142 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:13.154323 3142 heartbeater.cc:511] Master 127.0.97.62:42715 requested a full tablet report, sending...
I20260929 20:53:13.155814 2819 ts_manager.cc:194] Registered new tserver with Master: 8499f1e527244cb7a90083d9b5624505 (127.0.97.2:41849)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:53:13.156603 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:53:13.177179 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42715
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:13.475044 3147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:13.475301 3147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:13.475461 3147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:13.482504 3147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:13.482604 3147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:13.482718 3147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:53:13.492128 3147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:13.493301 3147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:13.494541 3147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:13.506976 3152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:13.506920 3153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:13.506922 3155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:13.507804 3147 server_base.cc:1034] running on GCE node
I20260929 20:53:13.508366 3147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:13.509250 3147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:13.510439 3147 hybrid_clock.cc:648] HybridClock initialized: now 1790715193510375 us; error 60 us; skew 500 ppm
I20260929 20:53:13.510736 3147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:13 dist-test-slave-m9g1 krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:13.513621 3147 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:53:13.514950 3147 webserver.cc:460] Webserver started at http://127.0.97.3:35003/ using document root <none> and password file <none>
I20260929 20:53:13.515579 3147 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:13.515676 3147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:13.515928 3147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:13.518728 3147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "9b550839cff84999a371de7d42d10851"
format_stamp: "Formatted at 2026-09-29 20:53:13 on dist-test-slave-m9g1"
server_key: "b783c2ba954f8985dc5b8ac44c3e0bbd"
server_key_iv: "f3b3aaba984c4aee644e44826c73fd6b"
server_key_version: "encryptionkey@0"
I20260929 20:53:13.519439 3147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9b550839cff84999a371de7d42d10851"
format_stamp: "Formatted at 2026-09-29 20:53:13 on dist-test-slave-m9g1"
server_key: "b783c2ba954f8985dc5b8ac44c3e0bbd"
server_key_iv: "f3b3aaba984c4aee644e44826c73fd6b"
server_key_version: "encryptionkey@0"
I20260929 20:53:13.524235 3147 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:53:13.527508 3162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:13.528764 3147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:53:13.528888 3147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "9b550839cff84999a371de7d42d10851"
format_stamp: "Formatted at 2026-09-29 20:53:13 on dist-test-slave-m9g1"
server_key: "b783c2ba954f8985dc5b8ac44c3e0bbd"
server_key_iv: "f3b3aaba984c4aee644e44826c73fd6b"
server_key_version: "encryptionkey@0"
I20260929 20:53:13.529047 3147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:13.546515 3147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:13.549929 3147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:13.550158 3147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:13.551002 3147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:13.552188 3147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:13.552254 3147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:13.552353 3147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:13.552394 3147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:13.579994 3147 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40311
I20260929 20:53:13.580034 3274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40311 every 8 connection(s)
I20260929 20:53:13.581204 3147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:53:13.581890 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 3147
I20260929 20:53:13.582065 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:53:13.582580 388 external_mini_cluster.cc:1428] Setting key 9da9e890bf65a3aff671a0ee66142197
Sep 29 20:53:13 dist-test-slave-m9g1 krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:13.599404 3010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:13.583900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54035 (local address 127.0.97.62:42715)
0929 20:53:13.584060 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:13.584064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:13.585329 (+ 1265us) server_negotiation.cc:407] Connection header received
0929 20:53:13.587097 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:13.587106 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:13.587166 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:13.587236 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:13.588554 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:13.589224 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:13.590654 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:13.591050 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:13.591142 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:13.594111 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:13.594116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:13.594120 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:13.594150 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:13.596620 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:13.597316 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:13.597321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:13.597325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:13.597468 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:13.597838 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:13.597843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:13.597847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:13.598109 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:13.598257 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:13.598884 (+ 627us) server_negotiation.cc:299] Negotiation successful
0929 20:53:13.599079 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 20:53:13.600333 3277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:13.584410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42715 (local address 127.0.97.3:54035)
0929 20:53:13.585159 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:13.585218 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:13.586866 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:13.587417 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:13.587425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:13.587903 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:13.588436 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:13.588452 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:13.589331 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:13.589334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:13.590557 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:13.590568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:13.591214 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:13.591219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:13.591394 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:13.591972 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:13.591996 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:13.593908 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:13.596734 (+ 2826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:13.596740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:13.596751 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:13.597213 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:13.597558 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:13.597563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:13.597567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:13.597736 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:13.598250 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:13.598258 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:13.598555 (+ 297us) client_negotiation.cc:769] Sending connection context
0929 20:53:13.598787 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 20:53:13.599274 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":310,"threads_started":1}
I20260929 20:53:13.601473 3275 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42715
I20260929 20:53:13.601732 3275 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:13.602447 3275 heartbeater.cc:511] Master 127.0.97.62:42715 requested a full tablet report, sending...
I20260929 20:53:13.603813 2819 ts_manager.cc:194] Registered new tserver with Master: 9b550839cff84999a371de7d42d10851 (127.0.97.3:40311)
I20260929 20:53:13.604597 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54035
I20260929 20:53:13.609220 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 20:53:13.635015 3010 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:13.632859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47798 (local address 127.0.97.62:42715)
0929 20:53:13.633063 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:13.633070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:13.633391 (+ 321us) server_negotiation.cc:407] Connection header received
0929 20:53:13.633506 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:13.633510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:13.633582 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:13.633673 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:13.634774 (+ 1101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47798: BlockingRecv error: recv got EOF from 127.0.0.1:47798 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:53:13.637779 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 2880
W20260929 20:53:13.639968 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 2880: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:42201: server requires authentication, but client does not have Kerberos credentials available
I20260929 20:53:13.640126 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 2880
I20260929 20:53:13.651841 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 3014
W20260929 20:53:13.653983 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 3014: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:41849: server requires authentication, but client does not have Kerberos credentials available
I20260929 20:53:13.654111 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 3014
I20260929 20:53:13.666566 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 3147
W20260929 20:53:13.669288 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 3147: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:40311: server requires authentication, but client does not have Kerberos credentials available
I20260929 20:53:13.669401 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 3147
I20260929 20:53:13.680912 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 2787
I20260929 20:53:13.681932 2849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:13.779790 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 2787
2026-09-29T20:53:13Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3189 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:53:13 dist-test-slave-m9g1 krb5kdc[3292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:53:13 dist-test-slave-m9g1 krb5kdc[3292](info): set up 2 sockets
Sep 29 20:53:13 dist-test-slave-m9g1 krb5kdc[3292](info): commencing operation
krb5kdc: starting...
W20260929 20:53:14.774394 388 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:53:14 dist-test-slave-m9g1 krb5kdc[3292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:53:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:53:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:53:14.943759 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41255
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:15.244447 3305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:15.244720 3305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:15.244836 3305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:15.251811 3305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:53:15.251909 3305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:15.251981 3305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:15.252058 3305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:53:15.252120 3305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:53:15.261685 3305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:15.262871 3305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:15.264120 3305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:15.275003 3310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:15.275075 3313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:15.275580 3311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:15.275554 3305 server_base.cc:1034] running on GCE node
I20260929 20:53:15.276330 3305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:15.277573 3305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:15.278755 3305 hybrid_clock.cc:648] HybridClock initialized: now 1790715195278719 us; error 40 us; skew 500 ppm
I20260929 20:53:15.279062 3305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:15 dist-test-slave-m9g1 krb5kdc[3292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:15.281862 3305 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:53:15.283159 3305 webserver.cc:460] Webserver started at http://127.0.97.62:40099/ using document root <none> and password file <none>
I20260929 20:53:15.283773 3305 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:15.283883 3305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:15.284143 3305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:15.286940 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "b289be1ac4fb4b6187591e782080d8f0"
format_stamp: "Formatted at 2026-09-29 20:53:15 on dist-test-slave-m9g1"
server_key: "6e09755a4131d0b4d817a043a1af0af1"
server_key_iv: "8b741d6d05faef7e6120a9da163caf72"
server_key_version: "encryptionkey@0"
I20260929 20:53:15.287684 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "b289be1ac4fb4b6187591e782080d8f0"
format_stamp: "Formatted at 2026-09-29 20:53:15 on dist-test-slave-m9g1"
server_key: "6e09755a4131d0b4d817a043a1af0af1"
server_key_iv: "8b741d6d05faef7e6120a9da163caf72"
server_key_version: "encryptionkey@0"
I20260929 20:53:15.292546 3305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:53:15.295682 3320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:15.296917 3305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:53:15.297043 3305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "b289be1ac4fb4b6187591e782080d8f0"
format_stamp: "Formatted at 2026-09-29 20:53:15 on dist-test-slave-m9g1"
server_key: "6e09755a4131d0b4d817a043a1af0af1"
server_key_iv: "8b741d6d05faef7e6120a9da163caf72"
server_key_version: "encryptionkey@0"
I20260929 20:53:15.297194 3305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:15.311506 3305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:15.315316 3305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:15.315536 3305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:15.330497 3305 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41255
I20260929 20:53:15.330535 3371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41255 every 8 connection(s)
I20260929 20:53:15.331670 3305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:53:15.334923 3372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:15.338850 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 3305
I20260929 20:53:15.339040 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:53:15.339797 388 external_mini_cluster.cc:1428] Setting key 44235f706b1bfa9ef23d8a698b8520db
I20260929 20:53:15.345764 3372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0: Bootstrap starting.
I20260929 20:53:15.349107 3372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:15.350219 3372 log.cc:824] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:15.353122 3372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0: No bootstrap required, opened a new log
Sep 29 20:53:15 dist-test-slave-m9g1 krb5kdc[3292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:15.360421 3372 raft_consensus.cc:348] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b289be1ac4fb4b6187591e782080d8f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41255 } }
I20260929 20:53:15.360764 3372 raft_consensus.cc:374] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:15.360854 3372 raft_consensus.cc:729] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b289be1ac4fb4b6187591e782080d8f0, State: Initialized, Role: FOLLOWER
I20260929 20:53:15.361296 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:15.341530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.0.97.62:41255)
0929 20:53:15.342345 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:15.342357 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:15.342397 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:53:15.344313 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:15.344334 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:15.344701 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:15.345018 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:15.346732 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:15.348029 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:15.349683 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:15.350397 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:15.350513 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:15.354692 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:15.354700 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:15.354714 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:15.354751 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:15.357531 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:15.358119 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:15.358125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:15.358131 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:15.358320 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:15.358651 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:15.358656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:15.358660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:15.359188 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:15.359455 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:15.359852 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 20:53:15.360238 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":406,"threads_started":1}
I20260929 20:53:15.361397 3372 consensus_queue.cc:260] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b289be1ac4fb4b6187591e782080d8f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41255 } }
I20260929 20:53:15.361629 3372 raft_consensus.cc:388] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:53:15.361759 3372 raft_consensus.cc:482] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:53:15.361939 3372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:15.364315 3372 raft_consensus.cc:504] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b289be1ac4fb4b6187591e782080d8f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41255 } }
I20260929 20:53:15.364774 3372 leader_election.cc:302] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b289be1ac4fb4b6187591e782080d8f0; no voters:
I20260929 20:53:15.364949 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:53:15.365397 3372 leader_election.cc:288] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:53:15.365492 3377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:15.367590 3372 sys_catalog.cc:564] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:53:15.373234 3377 raft_consensus.cc:686] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0 [term 1 LEADER]: Becoming Leader. State: Replica: b289be1ac4fb4b6187591e782080d8f0, State: Running, Role: LEADER
I20260929 20:53:15.373723 3377 consensus_queue.cc:237] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b289be1ac4fb4b6187591e782080d8f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41255 } }
I20260929 20:53:15.376751 3378 sys_catalog.cc:455] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b289be1ac4fb4b6187591e782080d8f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b289be1ac4fb4b6187591e782080d8f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41255 } } }
I20260929 20:53:15.377061 3378 sys_catalog.cc:458] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:15.381899 3379 sys_catalog.cc:455] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b289be1ac4fb4b6187591e782080d8f0. Latest consensus state: current_term: 1 leader_uuid: "b289be1ac4fb4b6187591e782080d8f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b289be1ac4fb4b6187591e782080d8f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41255 } } }
I20260929 20:53:15.382685 3379 sys_catalog.cc:458] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0 [sys.catalog]: This master's current role is: LEADER
W20260929 20:53:15.389219 3392 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:53:15.389350 3392 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:53:15.389509 3393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:53:15.391325 3393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:53:15.399837 3393 catalog_manager.cc:1347] Generated new cluster ID: cbd3417f76c947798002a9457d84a3d7
I20260929 20:53:15.399937 3393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:53:15.408080 3393 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:53:15.408864 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:15.396881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33406 (local address 127.0.97.62:41255)
0929 20:53:15.397046 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:15.397052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:15.397074 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:53:15.397131 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:15.397136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:15.397211 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:15.397329 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:15.398446 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:15.399186 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:15.400924 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:15.401539 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:15.401643 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:15.407219 (+ 5576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:15.407227 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:15.407232 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:53:15.408013 (+ 781us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:15.408110 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:53:15.408145 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:15.408247 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:15.408451 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 20:53:15.408541 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:53:15.409719 3393 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:53:15.418788 3393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b289be1ac4fb4b6187591e782080d8f0: Generated new TSK 0
I20260929 20:53:15.419771 3393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:53:15.434279 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 3305
I20260929 20:53:15.435280 3367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:15.521780 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 3305
2026-09-29T20:53:15Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1737 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:53:15 dist-test-slave-m9g1 krb5kdc[3406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:53:15 dist-test-slave-m9g1 krb5kdc[3406](info): set up 2 sockets
Sep 29 20:53:15 dist-test-slave-m9g1 krb5kdc[3406](info): commencing operation
krb5kdc: starting...
W20260929 20:53:16.502429 388 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:53:16 dist-test-slave-m9g1 krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:53:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:53:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:53:16.677021 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36255
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:16.980669 3419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:16.980949 3419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:16.981073 3419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:16.988133 3419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:53:16.988240 3419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:16.988309 3419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:16.988384 3419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:53:16.988451 3419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:53:16.998006 3419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:16.999280 3419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:17.000572 3419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:17.011343 3424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:17.011370 3425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:17.011601 3427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:17.012034 3419 server_base.cc:1034] running on GCE node
I20260929 20:53:17.012800 3419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:17.014128 3419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:17.015333 3419 hybrid_clock.cc:648] HybridClock initialized: now 1790715197015273 us; error 53 us; skew 500 ppm
I20260929 20:53:17.015630 3419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:17 dist-test-slave-m9g1 krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:17.018464 3419 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:53:17.019768 3419 webserver.cc:460] Webserver started at http://127.0.97.62:42063/ using document root <none> and password file <none>
I20260929 20:53:17.020393 3419 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:17.020488 3419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:17.020722 3419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:17.023582 3419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "51b1015f38c64e54a10b8969fd78b262"
format_stamp: "Formatted at 2026-09-29 20:53:17 on dist-test-slave-m9g1"
server_key: "d98cecd94f91ef05bc335b8a50983d63"
server_key_iv: "b380ee3095aee18837874aaeb4ac9f1d"
server_key_version: "encryptionkey@0"
I20260929 20:53:17.024260 3419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "51b1015f38c64e54a10b8969fd78b262"
format_stamp: "Formatted at 2026-09-29 20:53:17 on dist-test-slave-m9g1"
server_key: "d98cecd94f91ef05bc335b8a50983d63"
server_key_iv: "b380ee3095aee18837874aaeb4ac9f1d"
server_key_version: "encryptionkey@0"
I20260929 20:53:17.029084 3419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:53:17.032374 3434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:17.033639 3419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:53:17.033776 3419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "51b1015f38c64e54a10b8969fd78b262"
format_stamp: "Formatted at 2026-09-29 20:53:17 on dist-test-slave-m9g1"
server_key: "d98cecd94f91ef05bc335b8a50983d63"
server_key_iv: "b380ee3095aee18837874aaeb4ac9f1d"
server_key_version: "encryptionkey@0"
I20260929 20:53:17.033939 3419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:17.047525 3419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:17.051492 3419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:17.051725 3419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:17.068392 3419 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36255
I20260929 20:53:17.068413 3485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36255 every 8 connection(s)
I20260929 20:53:17.069684 3419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:53:17.073261 3486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:17.073560 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 3419
I20260929 20:53:17.073725 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:53:17.074375 388 external_mini_cluster.cc:1428] Setting key f3a6c6f365bbc52f961971a07ab21749
I20260929 20:53:17.086315 3486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262: Bootstrap starting.
Sep 29 20:53:17 dist-test-slave-m9g1 krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:17.089656 3486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:17.090795 3486 log.cc:824] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:17.093423 3486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262: No bootstrap required, opened a new log
I20260929 20:53:17.096252 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:17.076330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.0.97.62:36255)
0929 20:53:17.077081 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:17.077102 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:17.077143 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:53:17.078887 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:17.078917 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:17.079322 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:17.079642 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:17.081233 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:17.082544 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:17.084234 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:17.084894 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:17.085023 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:17.089282 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:17.089290 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:17.089309 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:17.089347 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:17.092204 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:17.092796 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:17.092804 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:17.092810 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:17.092975 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:17.093315 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:17.093320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:17.093323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:17.093836 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:17.094127 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:17.094535 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 20:53:17.094940 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":313,"threads_started":1}
I20260929 20:53:17.101202 3486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b1015f38c64e54a10b8969fd78b262" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36255 } }
I20260929 20:53:17.101495 3486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:17.101562 3486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b1015f38c64e54a10b8969fd78b262, State: Initialized, Role: FOLLOWER
I20260929 20:53:17.101999 3486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b1015f38c64e54a10b8969fd78b262" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36255 } }
I20260929 20:53:17.102171 3486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:53:17.102308 3486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:53:17.102422 3486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:17.104455 3486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b1015f38c64e54a10b8969fd78b262" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36255 } }
I20260929 20:53:17.104876 3486 leader_election.cc:302] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51b1015f38c64e54a10b8969fd78b262; no voters:
I20260929 20:53:17.105392 3486 leader_election.cc:288] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:53:17.105500 3491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:17.106105 3491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262 [term 1 LEADER]: Becoming Leader. State: Replica: 51b1015f38c64e54a10b8969fd78b262, State: Running, Role: LEADER
I20260929 20:53:17.106613 3491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b1015f38c64e54a10b8969fd78b262" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36255 } }
I20260929 20:53:17.107167 3486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:53:17.108994 3492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51b1015f38c64e54a10b8969fd78b262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b1015f38c64e54a10b8969fd78b262" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36255 } } }
I20260929 20:53:17.109361 3492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:17.108971 3493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51b1015f38c64e54a10b8969fd78b262. Latest consensus state: current_term: 1 leader_uuid: "51b1015f38c64e54a10b8969fd78b262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b1015f38c64e54a10b8969fd78b262" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36255 } } }
I20260929 20:53:17.109719 3493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:17.111013 3496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:53:17.115742 3496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:53:17.124223 3496 catalog_manager.cc:1347] Generated new cluster ID: 4b6b39555684448abb77b9157f47f14a
I20260929 20:53:17.124333 3496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:53:17.145780 3496 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:53:17.147109 3496 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:53:17.155028 3496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51b1015f38c64e54a10b8969fd78b262: Generated new TSK 0
I20260929 20:53:17.155822 3496 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:53:17.203852 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36255
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:17.519182 3512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:17.519454 3512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:17.519567 3512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:17.526777 3512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:17.526887 3512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:17.527031 3512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:53:17.536839 3512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:17.538153 3512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:17.539418 3512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:17.551578 3517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:17.551642 3520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:17.551612 3518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:17.552408 3512 server_base.cc:1034] running on GCE node
I20260929 20:53:17.552999 3512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:17.553871 3512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:17.555063 3512 hybrid_clock.cc:648] HybridClock initialized: now 1790715197555009 us; error 46 us; skew 500 ppm
I20260929 20:53:17.555351 3512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:17 dist-test-slave-m9g1 krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:17.558141 3512 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:53:17.559484 3512 webserver.cc:460] Webserver started at http://127.0.97.1:39855/ using document root <none> and password file <none>
I20260929 20:53:17.560142 3512 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:17.560250 3512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:17.560549 3512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:17.563338 3512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "8ccdf086947b4f648486895fc196b634"
format_stamp: "Formatted at 2026-09-29 20:53:17 on dist-test-slave-m9g1"
server_key: "6980960e4847a0914e883d608ddb6d6a"
server_key_iv: "691ca31880b02746564c3cae34d16268"
server_key_version: "encryptionkey@0"
I20260929 20:53:17.564061 3512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8ccdf086947b4f648486895fc196b634"
format_stamp: "Formatted at 2026-09-29 20:53:17 on dist-test-slave-m9g1"
server_key: "6980960e4847a0914e883d608ddb6d6a"
server_key_iv: "691ca31880b02746564c3cae34d16268"
server_key_version: "encryptionkey@0"
I20260929 20:53:17.568786 3512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:53:17.572029 3527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:17.573272 3512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:53:17.573406 3512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "8ccdf086947b4f648486895fc196b634"
format_stamp: "Formatted at 2026-09-29 20:53:17 on dist-test-slave-m9g1"
server_key: "6980960e4847a0914e883d608ddb6d6a"
server_key_iv: "691ca31880b02746564c3cae34d16268"
server_key_version: "encryptionkey@0"
I20260929 20:53:17.573575 3512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:17.587749 3512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:17.591631 3512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:17.591900 3512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:17.593092 3512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:17.594472 3512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:17.594540 3512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:17.594630 3512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:17.594681 3512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:17.623349 3512 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39085
I20260929 20:53:17.623390 3639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39085 every 8 connection(s)
I20260929 20:53:17.624583 3512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:53:17.631767 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 3512
I20260929 20:53:17.631928 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:53:17.632521 388 external_mini_cluster.cc:1428] Setting key 43aabc24626d8abb64a2174aa7f14740
Sep 29 20:53:17 dist-test-slave-m9g1 krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:17.643400 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:17.626943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43127 (local address 127.0.97.62:36255)
0929 20:53:17.627330 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:17.627334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:17.628268 (+ 934us) server_negotiation.cc:407] Connection header received
0929 20:53:17.629959 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:17.629964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:17.630046 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:17.630137 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:17.631604 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:17.632393 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:17.634256 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:17.634637 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:17.634753 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:17.638101 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:17.638106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:17.638109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:17.638144 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:17.640480 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:17.641272 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:17.641276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:17.641278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:17.641381 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:17.641825 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:17.641829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:17.641831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:17.642110 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:17.642271 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:17.642989 (+ 718us) server_negotiation.cc:299] Negotiation successful
0929 20:53:17.643154 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":195,"threads_started":1}
I20260929 20:53:17.644781 3643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:17.627469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36255 (local address 127.0.97.1:43127)
0929 20:53:17.628123 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:17.628159 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:17.629706 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:17.630397 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:17.630406 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:17.630887 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:17.631440 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:17.631457 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:17.632538 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:17.632543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:17.633883 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:17.633895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:17.634850 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:17.634858 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:17.635093 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:17.635744 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:17.635766 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:17.637893 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:17.640609 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:17.640618 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:17.640635 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:17.641171 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:17.641461 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:17.641465 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:17.641468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:17.641690 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:17.642253 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:17.642260 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:17.642649 (+ 389us) client_negotiation.cc:769] Sending connection context
0929 20:53:17.642923 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 20:53:17.643438 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":257,"threads_started":1}
I20260929 20:53:17.646327 3640 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36255
I20260929 20:53:17.646608 3640 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:17.647296 3640 heartbeater.cc:511] Master 127.0.97.62:36255 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:53:17.649304 3451 ts_manager.cc:194] Registered new tserver with Master: 8ccdf086947b4f648486895fc196b634 (127.0.97.1:39085)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 20:53:17.650942 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:53:17.671154 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36255
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:17.972007 3646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:17.972280 3646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:17.972420 3646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:17.979418 3646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:17.979538 3646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:17.979687 3646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:53:17.988999 3646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:17.990252 3646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:17.991480 3646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:18.003687 3652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:18.003774 3651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:18.003752 3654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:18.004535 3646 server_base.cc:1034] running on GCE node
I20260929 20:53:18.005167 3646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:18.006155 3646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:18.007386 3646 hybrid_clock.cc:648] HybridClock initialized: now 1790715198007315 us; error 67 us; skew 500 ppm
I20260929 20:53:18.007687 3646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:18 dist-test-slave-m9g1 krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:18.010805 3646 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:53:18.012226 3646 webserver.cc:460] Webserver started at http://127.0.97.2:38687/ using document root <none> and password file <none>
I20260929 20:53:18.012943 3646 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:18.013054 3646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:18.013329 3646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:18.016217 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "bf6afef4fb744f3e83f41926889dc5a1"
format_stamp: "Formatted at 2026-09-29 20:53:18 on dist-test-slave-m9g1"
server_key: "c6927156ce222735f7db5c914736cff7"
server_key_iv: "afd3e239746b134574e2c4b8a89a8dd0"
server_key_version: "encryptionkey@0"
I20260929 20:53:18.016947 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bf6afef4fb744f3e83f41926889dc5a1"
format_stamp: "Formatted at 2026-09-29 20:53:18 on dist-test-slave-m9g1"
server_key: "c6927156ce222735f7db5c914736cff7"
server_key_iv: "afd3e239746b134574e2c4b8a89a8dd0"
server_key_version: "encryptionkey@0"
I20260929 20:53:18.021879 3646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 20:53:18.025355 3661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:18.026726 3646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:53:18.026873 3646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "bf6afef4fb744f3e83f41926889dc5a1"
format_stamp: "Formatted at 2026-09-29 20:53:18 on dist-test-slave-m9g1"
server_key: "c6927156ce222735f7db5c914736cff7"
server_key_iv: "afd3e239746b134574e2c4b8a89a8dd0"
server_key_version: "encryptionkey@0"
I20260929 20:53:18.027041 3646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:18.043402 3646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:18.047483 3646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:18.047781 3646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:18.048776 3646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:18.050050 3646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:18.050134 3646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:18.050274 3646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:18.050334 3646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:18.086175 3646 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39745
I20260929 20:53:18.086254 3773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39745 every 8 connection(s)
I20260929 20:53:18.087455 3646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:53:18.096328 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 3646
I20260929 20:53:18.096498 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:53:18.097013 388 external_mini_cluster.cc:1428] Setting key ecb85b7ce4080d1fddf176bb6d1ce5dd
Sep 29 20:53:18 dist-test-slave-m9g1 krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:18.105969 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:18.089682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53121 (local address 127.0.97.62:36255)
0929 20:53:18.089877 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:18.089883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:18.091068 (+ 1185us) server_negotiation.cc:407] Connection header received
0929 20:53:18.092707 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:18.092711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:18.092775 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:18.092855 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:18.094222 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:18.094895 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:18.096326 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:18.096922 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:18.097057 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:18.100450 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:18.100456 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:18.100467 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:18.100504 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:18.103260 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:18.103862 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:18.103867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:18.103870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:18.104019 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:18.104486 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:18.104490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:18.104492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:18.104744 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:18.104860 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:18.105593 (+ 733us) server_negotiation.cc:299] Negotiation successful
0929 20:53:18.105748 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:53:18.107331 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:18.090262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36255 (local address 127.0.97.2:53121)
0929 20:53:18.090923 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:18.090959 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:18.092474 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:18.093055 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:18.093062 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:18.093529 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:18.094066 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:18.094084 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:18.095010 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:18.095014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:18.096197 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:18.096208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:18.097133 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:18.097141 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:18.097331 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:18.097951 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:18.097976 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:18.100247 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:18.103372 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:18.103379 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:18.103390 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:18.103766 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:18.104123 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:18.104128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:18.104132 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:18.104345 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:18.104864 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:18.104872 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:18.105273 (+ 401us) client_negotiation.cc:769] Sending connection context
0929 20:53:18.105542 (+ 269us) client_negotiation.cc:240] Negotiation successful
0929 20:53:18.106021 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":248,"threads_started":1}
I20260929 20:53:18.108668 3774 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36255
I20260929 20:53:18.108949 3774 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:18.109730 3774 heartbeater.cc:511] Master 127.0.97.62:36255 requested a full tablet report, sending...
I20260929 20:53:18.111151 3451 ts_manager.cc:194] Registered new tserver with Master: bf6afef4fb744f3e83f41926889dc5a1 (127.0.97.2:39745)
I20260929 20:53:18.112231 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53121
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:53:18.136631 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36255
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:18.437570 3779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:18.437868 3779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:18.438020 3779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:18.445031 3779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:18.445133 3779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:18.445287 3779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:53:18.454582 3779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:18.455770 3779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:18.456986 3779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:18.468881 3784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:18.468909 3785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:18.468842 3787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:18.469710 3779 server_base.cc:1034] running on GCE node
I20260929 20:53:18.470238 3779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:18.471051 3779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:18.472231 3779 hybrid_clock.cc:648] HybridClock initialized: now 1790715198472193 us; error 29 us; skew 500 ppm
I20260929 20:53:18.472479 3779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:18 dist-test-slave-m9g1 krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:18.475272 3779 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:53:18.476533 3779 webserver.cc:460] Webserver started at http://127.0.97.3:44383/ using document root <none> and password file <none>
I20260929 20:53:18.477164 3779 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:18.477255 3779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:18.477527 3779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:18.480291 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "b562cea14d714ce6b6ea82df0aa27853"
format_stamp: "Formatted at 2026-09-29 20:53:18 on dist-test-slave-m9g1"
server_key: "c8fa5271c7dce0487704e6c4c79d6aba"
server_key_iv: "effb31f3f9fa6a65ed74fb3a71cf0f52"
server_key_version: "encryptionkey@0"
I20260929 20:53:18.481011 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b562cea14d714ce6b6ea82df0aa27853"
format_stamp: "Formatted at 2026-09-29 20:53:18 on dist-test-slave-m9g1"
server_key: "c8fa5271c7dce0487704e6c4c79d6aba"
server_key_iv: "effb31f3f9fa6a65ed74fb3a71cf0f52"
server_key_version: "encryptionkey@0"
I20260929 20:53:18.485948 3779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:53:18.489132 3794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:18.490525 3779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:53:18.490666 3779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "b562cea14d714ce6b6ea82df0aa27853"
format_stamp: "Formatted at 2026-09-29 20:53:18 on dist-test-slave-m9g1"
server_key: "c8fa5271c7dce0487704e6c4c79d6aba"
server_key_iv: "effb31f3f9fa6a65ed74fb3a71cf0f52"
server_key_version: "encryptionkey@0"
I20260929 20:53:18.490839 3779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:18.503065 3779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:18.506546 3779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:18.506788 3779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:18.507664 3779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:18.508891 3779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:18.508956 3779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:18.509038 3779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:18.509078 3779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:18.537220 3779 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37479
I20260929 20:53:18.537262 3906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37479 every 8 connection(s)
I20260929 20:53:18.538486 3779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:53:18.540350 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 3779
I20260929 20:53:18.540517 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:53:18.541083 388 external_mini_cluster.cc:1428] Setting key e2d0785bedf6ca625d2ecceeedb74090
Sep 29 20:53:18 dist-test-slave-m9g1 krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:18.563002 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:18.541826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58021 (local address 127.0.97.62:36255)
0929 20:53:18.542066 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:18.542077 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:18.543176 (+ 1099us) server_negotiation.cc:407] Connection header received
0929 20:53:18.545781 (+ 2605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:18.545787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:18.545873 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:18.545983 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:18.547910 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:18.548858 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:18.550831 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:18.551624 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:18.551742 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:18.555728 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:18.555735 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:18.555741 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:18.555778 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:18.559484 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:18.560369 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:18.560374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:18.560379 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:18.560506 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:18.561050 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:18.561055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:18.561059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:18.561427 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:18.561571 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:18.562429 (+ 858us) server_negotiation.cc:299] Negotiation successful
0929 20:53:18.562646 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:53:18.564127 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:18.541786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36255 (local address 127.0.97.3:58021)
0929 20:53:18.543028 (+ 1242us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:18.543078 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:18.545403 (+ 2325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:18.546277 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:18.546290 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:18.547003 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:18.547734 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:18.547756 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:18.549026 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:18.549031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:18.550686 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:18.550700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:18.551840 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:18.551846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:18.552092 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:18.552707 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:18.552737 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:18.555331 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:18.559617 (+ 4286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:18.559627 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:18.559647 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:18.560254 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:18.560607 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:18.560613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:18.560616 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:18.560900 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:18.561564 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:18.561573 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:18.561960 (+ 387us) client_negotiation.cc:769] Sending connection context
0929 20:53:18.562299 (+ 339us) client_negotiation.cc:240] Negotiation successful
0929 20:53:18.562831 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"thread_start_us":898,"threads_started":1}
I20260929 20:53:18.565994 3907 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36255
I20260929 20:53:18.566367 3907 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:18.567257 3907 heartbeater.cc:511] Master 127.0.97.62:36255 requested a full tablet report, sending...
I20260929 20:53:18.568769 3451 ts_manager.cc:194] Registered new tserver with Master: b562cea14d714ce6b6ea82df0aa27853 (127.0.97.3:37479)
I20260929 20:53:18.569693 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58021
I20260929 20:53:18.571681 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:53:18 dist-test-slave-m9g1 krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715198, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 20:53:18 dist-test-slave-m9g1 krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715198, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:18.619400 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:18.606108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58082 (local address 127.0.97.62:36255)
0929 20:53:18.606300 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:18.606306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:18.606675 (+ 369us) server_negotiation.cc:407] Connection header received
0929 20:53:18.606849 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:18.606853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:18.606924 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:18.607008 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:18.608480 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:18.609324 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:18.610565 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:18.610947 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:18.611053 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:18.614573 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:18.614577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:18.614581 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:18.614614 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:18.616892 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:18.617504 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:18.617508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:18.617510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:18.617643 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:18.618023 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:18.618026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:18.618028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:18.618372 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:18.618599 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:18.618895 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 20:53:18.619113 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 20:53:18.619946 3451 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58082
I20260929 20:53:18.622143 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 3512
Sep 29 20:53:18 dist-test-slave-m9g1 krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715198, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:53:18.637674 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:18.623240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35438 (local address 127.0.97.1:39085)
0929 20:53:18.623837 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:18.623844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:18.623866 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:53:18.623937 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:18.623943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:18.624141 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:18.624304 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:18.625437 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:18.626537 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:18.627707 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:18.628257 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:18.628368 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:18.632467 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:18.632479 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:18.632493 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:18.632527 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:18.635056 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:18.635694 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:18.635701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:18.635703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:18.635825 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:18.636163 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:18.636167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:18.636169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:18.636498 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:18.636780 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:18.637077 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 20:53:18.637326 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":332,"threads_started":1}
W20260929 20:53:18.638942 3635 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35438
W20260929 20:53:18.639981 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 3512: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:53:18.640174 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 3512
I20260929 20:53:18.652582 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 3646
Sep 29 20:53:18 dist-test-slave-m9g1 krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715198, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:53:18.667097 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:18.653350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60882 (local address 127.0.97.2:39745)
0929 20:53:18.653869 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:18.653876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:18.653901 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:53:18.653968 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:18.653974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:18.654208 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:18.654349 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:18.655479 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:18.656452 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:18.657566 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:18.658088 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:18.658180 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:18.661822 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:18.661833 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:18.661847 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:18.661886 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:18.664504 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:18.665100 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:18.665106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:18.665109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:18.665229 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:18.665572 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:18.665576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:18.665578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:18.665928 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:18.666223 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:18.666515 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 20:53:18.666756 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":253,"threads_started":1}
W20260929 20:53:18.668249 3769 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60882
W20260929 20:53:18.669293 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 3646: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:53:18.669430 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 3646
I20260929 20:53:18.681257 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 3779
Sep 29 20:53:18 dist-test-slave-m9g1 krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715198, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:53:18.695642 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:18.682162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35892 (local address 127.0.97.3:37479)
0929 20:53:18.682763 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:18.682769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:18.682796 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:53:18.682854 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:18.682860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:18.683063 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:18.683181 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:18.684390 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:18.685392 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:18.686578 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:18.687098 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:18.687192 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:18.690922 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:18.690933 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:18.690948 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:18.690979 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:18.693292 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:18.693890 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:18.693897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:18.693899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:18.694007 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:18.694416 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:18.694419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:18.694422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:18.694726 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:18.695003 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:18.695159 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 20:53:18.695367 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":283,"threads_started":1}
W20260929 20:53:18.696801 3902 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35892
W20260929 20:53:18.697826 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 3779: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:53:18.697970 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 3779
I20260929 20:53:18.710220 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 3419
I20260929 20:53:18.711262 3481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:18.810279 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 3419
2026-09-29T20:53:18Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3296 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:53:18 dist-test-slave-m9g1 krb5kdc[3924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:53:18 dist-test-slave-m9g1 krb5kdc[3924](info): set up 2 sockets
Sep 29 20:53:18 dist-test-slave-m9g1 krb5kdc[3924](info): commencing operation
krb5kdc: starting...
W20260929 20:53:19.810236 388 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:53:19 dist-test-slave-m9g1 krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:53:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:53:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:53:19.983984 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32991
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:20.287783 3937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:20.288053 3937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:20.288177 3937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:20.295083 3937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:53:20.295184 3937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:20.295238 3937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:20.295311 3937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:53:20.295372 3937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:53:20.304759 3937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:20.305960 3937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:20.307228 3937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:20.318115 3942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:20.318074 3943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:20.318150 3945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:20.318828 3937 server_base.cc:1034] running on GCE node
I20260929 20:53:20.319622 3937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:20.320935 3937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:20.322134 3937 hybrid_clock.cc:648] HybridClock initialized: now 1790715200322073 us; error 46 us; skew 500 ppm
I20260929 20:53:20.322500 3937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:20 dist-test-slave-m9g1 krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:20.325315 3937 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:53:20.326671 3937 webserver.cc:460] Webserver started at http://127.0.97.62:36351/ using document root <none> and password file <none>
I20260929 20:53:20.327335 3937 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:20.327448 3937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:20.327735 3937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:20.330750 3937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "0d2c2b0a3ce942c28ce404bda2f11ea2"
format_stamp: "Formatted at 2026-09-29 20:53:20 on dist-test-slave-m9g1"
server_key: "bff0efe8729cb4d1aede524641a6d833"
server_key_iv: "63787cd2758e89e112cc4614d49b124a"
server_key_version: "encryptionkey@0"
I20260929 20:53:20.331485 3937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "0d2c2b0a3ce942c28ce404bda2f11ea2"
format_stamp: "Formatted at 2026-09-29 20:53:20 on dist-test-slave-m9g1"
server_key: "bff0efe8729cb4d1aede524641a6d833"
server_key_iv: "63787cd2758e89e112cc4614d49b124a"
server_key_version: "encryptionkey@0"
I20260929 20:53:20.336382 3937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:53:20.339648 3952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:20.340901 3937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:53:20.341055 3937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "0d2c2b0a3ce942c28ce404bda2f11ea2"
format_stamp: "Formatted at 2026-09-29 20:53:20 on dist-test-slave-m9g1"
server_key: "bff0efe8729cb4d1aede524641a6d833"
server_key_iv: "63787cd2758e89e112cc4614d49b124a"
server_key_version: "encryptionkey@0"
I20260929 20:53:20.341231 3937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:20.365871 3937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:20.369575 3937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:20.369822 3937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:20.385915 3937 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32991
I20260929 20:53:20.385946 4003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32991 every 8 connection(s)
I20260929 20:53:20.387236 3937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:53:20.389707 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 3937
I20260929 20:53:20.389902 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:53:20.390529 388 external_mini_cluster.cc:1428] Setting key 95dac5c258b69efb84f4786c6b8cf219
I20260929 20:53:20.390976 4004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:20.404379 4004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2: Bootstrap starting.
Sep 29 20:53:20 dist-test-slave-m9g1 krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:20.407495 4004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:20.408433 4004 log.cc:824] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:20.412262 4004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2: No bootstrap required, opened a new log
I20260929 20:53:20.416803 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:20.392336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38924 (local address 127.0.97.62:32991)
0929 20:53:20.393017 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:20.393029 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:20.393068 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:53:20.395305 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:20.395327 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:20.395719 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:20.396045 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:20.397697 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:20.399132 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:20.400833 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:20.401569 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:20.401710 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:20.405967 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:20.405975 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:20.405989 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:20.406024 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:20.412433 (+ 6409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:20.413102 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:20.413110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:20.413119 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:20.413315 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:20.413667 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:20.413671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:20.413674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:20.414384 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:20.414662 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:20.415009 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 20:53:20.415462 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":297,"threads_started":1}
I20260929 20:53:20.420883 4004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2c2b0a3ce942c28ce404bda2f11ea2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32991 } }
I20260929 20:53:20.421196 4004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:20.421309 4004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2c2b0a3ce942c28ce404bda2f11ea2, State: Initialized, Role: FOLLOWER
I20260929 20:53:20.421900 4004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2c2b0a3ce942c28ce404bda2f11ea2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32991 } }
I20260929 20:53:20.422088 4004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:53:20.422178 4004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:53:20.422334 4004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:20.424307 4004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2c2b0a3ce942c28ce404bda2f11ea2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32991 } }
I20260929 20:53:20.424717 4004 leader_election.cc:302] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d2c2b0a3ce942c28ce404bda2f11ea2; no voters:
I20260929 20:53:20.425333 4004 leader_election.cc:288] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:53:20.425420 4009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:20.426131 4009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2 [term 1 LEADER]: Becoming Leader. State: Replica: 0d2c2b0a3ce942c28ce404bda2f11ea2, State: Running, Role: LEADER
I20260929 20:53:20.426656 4009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2c2b0a3ce942c28ce404bda2f11ea2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32991 } }
I20260929 20:53:20.427372 4004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:53:20.429145 4010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d2c2b0a3ce942c28ce404bda2f11ea2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2c2b0a3ce942c28ce404bda2f11ea2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32991 } } }
I20260929 20:53:20.429121 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d2c2b0a3ce942c28ce404bda2f11ea2. Latest consensus state: current_term: 1 leader_uuid: "0d2c2b0a3ce942c28ce404bda2f11ea2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2c2b0a3ce942c28ce404bda2f11ea2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32991 } } }
I20260929 20:53:20.429529 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:20.429623 4010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:20.435164 4014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:53:20.439204 4014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:53:20.447279 4014 catalog_manager.cc:1347] Generated new cluster ID: 6300935088ed454c8307ae65847cd64a
I20260929 20:53:20.447387 4014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:53:20.488763 4014 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:53:20.490613 4014 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:53:20.499450 4014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d2c2b0a3ce942c28ce404bda2f11ea2: Generated new TSK 0
I20260929 20:53:20.500288 4014 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:53:20.553411 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32991
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:20.861267 4030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:20.861545 4030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:20.861675 4030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:20.868780 4030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:20.868880 4030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:20.869028 4030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:53:20.878384 4030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:20.879576 4030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:20.880813 4030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:20.892789 4036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:20.892825 4038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:20.892860 4035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:20.893671 4030 server_base.cc:1034] running on GCE node
I20260929 20:53:20.894225 4030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:20.895056 4030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:20.896276 4030 hybrid_clock.cc:648] HybridClock initialized: now 1790715200896227 us; error 38 us; skew 500 ppm
I20260929 20:53:20.896577 4030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:20 dist-test-slave-m9g1 krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:20.899413 4030 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:53:20.900698 4030 webserver.cc:460] Webserver started at http://127.0.97.1:36685/ using document root <none> and password file <none>
I20260929 20:53:20.901309 4030 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:20.901398 4030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:20.901646 4030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:20.904484 4030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "919edbc496f644a39c5fcb8721c02e5f"
format_stamp: "Formatted at 2026-09-29 20:53:20 on dist-test-slave-m9g1"
server_key: "4d7983eb20bbb35658025a72a26ad314"
server_key_iv: "fcf4ecb194f2dbf4bc764922594aa7b0"
server_key_version: "encryptionkey@0"
I20260929 20:53:20.905207 4030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "919edbc496f644a39c5fcb8721c02e5f"
format_stamp: "Formatted at 2026-09-29 20:53:20 on dist-test-slave-m9g1"
server_key: "4d7983eb20bbb35658025a72a26ad314"
server_key_iv: "fcf4ecb194f2dbf4bc764922594aa7b0"
server_key_version: "encryptionkey@0"
I20260929 20:53:20.910131 4030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260929 20:53:20.913554 4045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:20.914868 4030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:53:20.914999 4030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "919edbc496f644a39c5fcb8721c02e5f"
format_stamp: "Formatted at 2026-09-29 20:53:20 on dist-test-slave-m9g1"
server_key: "4d7983eb20bbb35658025a72a26ad314"
server_key_iv: "fcf4ecb194f2dbf4bc764922594aa7b0"
server_key_version: "encryptionkey@0"
I20260929 20:53:20.915164 4030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:20.933212 4030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:20.937093 4030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:20.937328 4030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:20.938172 4030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:20.939450 4030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:20.939527 4030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:20.939616 4030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:20.939657 4030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:20.967793 4030 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40163
I20260929 20:53:20.967840 4157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40163 every 8 connection(s)
I20260929 20:53:20.969020 4030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:53:20.971048 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 4030
I20260929 20:53:20.971230 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:53:20.971951 388 external_mini_cluster.cc:1428] Setting key 6753a9c10a91997c722870588840f93e
Sep 29 20:53:20 dist-test-slave-m9g1 krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:20.987958 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:20.971466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48487 (local address 127.0.97.62:32991)
0929 20:53:20.971899 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:20.971905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:20.972975 (+ 1070us) server_negotiation.cc:407] Connection header received
0929 20:53:20.974743 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:20.974749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:20.974822 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:20.974932 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:20.976277 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:20.977241 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:20.978744 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:20.979333 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:20.979446 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:20.982755 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:20.982760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:20.982763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:20.982793 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:20.985091 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:20.985737 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:20.985742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:20.985746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:20.985889 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:20.986273 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:20.986278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:20.986282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:20.986622 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:20.986769 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:20.987364 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 20:53:20.987585 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":204,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:53:20.988662 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:20.972039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32991 (local address 127.0.97.1:48487)
0929 20:53:20.972826 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:20.972874 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:20.974498 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:20.975119 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:20.975127 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:20.975605 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:20.976131 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:20.976148 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:20.977358 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:20.977361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:20.978617 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:20.978628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:20.979506 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:20.979511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:20.979696 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:20.980306 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:20.980330 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:20.982534 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:20.985213 (+ 2679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:20.985220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:20.985231 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:20.985611 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:20.985977 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:20.985980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:20.985982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:20.986152 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:20.986727 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:20.986732 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:20.987029 (+ 297us) client_negotiation.cc:769] Sending connection context
0929 20:53:20.987243 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 20:53:20.987601 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":304,"threads_started":1}
I20260929 20:53:20.990058 4158 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32991
I20260929 20:53:20.990365 4158 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 20:53:20.991135 4158 heartbeater.cc:511] Master 127.0.97.62:32991 requested a full tablet report, sending...
I20260929 20:53:20.993345 3969 ts_manager.cc:194] Registered new tserver with Master: 919edbc496f644a39c5fcb8721c02e5f (127.0.97.1:40163)
I20260929 20:53:20.995216 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:53:21.011155 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32991
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:21.318398 4164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:21.318657 4164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:21.318778 4164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:21.325835 4164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:21.325940 4164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:21.326057 4164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:53:21.335721 4164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:21.336944 4164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:21.338166 4164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:21.350656 4172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:21.350623 4169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:21.350862 4170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:21.351207 4164 server_base.cc:1034] running on GCE node
I20260929 20:53:21.351766 4164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:21.352631 4164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:21.353825 4164 hybrid_clock.cc:648] HybridClock initialized: now 1790715201353766 us; error 49 us; skew 500 ppm
I20260929 20:53:21.354125 4164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:21 dist-test-slave-m9g1 krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:21.356927 4164 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:53:21.358269 4164 webserver.cc:460] Webserver started at http://127.0.97.2:43675/ using document root <none> and password file <none>
I20260929 20:53:21.358942 4164 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:21.359055 4164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:21.359326 4164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:21.362085 4164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "73f63a7cd10e4501913369de756ed469"
format_stamp: "Formatted at 2026-09-29 20:53:21 on dist-test-slave-m9g1"
server_key: "c5862b66b61691f00097138d0c14d1ba"
server_key_iv: "07e4978295c8a0bada6de18acb9086fd"
server_key_version: "encryptionkey@0"
I20260929 20:53:21.362835 4164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "73f63a7cd10e4501913369de756ed469"
format_stamp: "Formatted at 2026-09-29 20:53:21 on dist-test-slave-m9g1"
server_key: "c5862b66b61691f00097138d0c14d1ba"
server_key_iv: "07e4978295c8a0bada6de18acb9086fd"
server_key_version: "encryptionkey@0"
I20260929 20:53:21.367560 4164 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:53:21.370683 4179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:21.371866 4164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 20:53:21.372004 4164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "73f63a7cd10e4501913369de756ed469"
format_stamp: "Formatted at 2026-09-29 20:53:21 on dist-test-slave-m9g1"
server_key: "c5862b66b61691f00097138d0c14d1ba"
server_key_iv: "07e4978295c8a0bada6de18acb9086fd"
server_key_version: "encryptionkey@0"
I20260929 20:53:21.372169 4164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:21.389133 4164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:21.393002 4164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:21.393244 4164 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260929 20:53:21.393934 4000 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.97.62:32991
I20260929 20:53:21.394147 4164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:21.395422 4164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:21.395494 4164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:21.395608 4164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:21.395658 4164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:21.424041 4164 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46399
I20260929 20:53:21.424074 4291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46399 every 8 connection(s)
I20260929 20:53:21.425239 4164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:53:21.427217 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 4164
I20260929 20:53:21.427420 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:53:21.427985 388 external_mini_cluster.cc:1428] Setting key efac014c9c3cbbda2abd39a7263efb90
Sep 29 20:53:21 dist-test-slave-m9g1 krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:53:21.445369 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:21.427754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52565 (local address 127.0.97.62:32991)
0929 20:53:21.427913 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:21.427917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:21.429294 (+ 1377us) server_negotiation.cc:407] Connection header received
0929 20:53:21.431190 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:21.431195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:21.431276 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:21.431379 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:21.433031 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:21.433972 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:21.435968 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:21.436354 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:21.436448 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:21.440092 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:21.440097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:21.440100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:21.440131 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:21.442614 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:21.443250 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:21.443253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:21.443255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:21.443384 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:21.443780 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:21.443784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:21.443786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:21.444060 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:21.444177 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:21.444843 (+ 666us) server_negotiation.cc:299] Negotiation successful
0929 20:53:21.445054 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:53:21.446336 4294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:21.428413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32991 (local address 127.0.97.2:52565)
0929 20:53:21.429150 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:21.429188 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:21.430926 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:21.431664 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:21.431674 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:21.432219 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:21.432837 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:21.432857 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:21.434100 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:21.434104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:21.435837 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:21.435851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:21.436569 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:21.436576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:21.436838 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:21.437499 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:21.437526 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:21.439889 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:21.442731 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:21.442741 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:21.442756 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:21.443136 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:21.443501 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:21.443506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:21.443510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:21.443676 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:21.444172 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:21.444179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:21.444492 (+ 313us) client_negotiation.cc:769] Sending connection context
0929 20:53:21.444716 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 20:53:21.445199 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":298,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 20:53:21.447706 4292 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32991
I20260929 20:53:21.447996 4292 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:21.448745 4292 heartbeater.cc:511] Master 127.0.97.62:32991 requested a full tablet report, sending...
I20260929 20:53:21.450212 3969 ts_manager.cc:194] Registered new tserver with Master: 73f63a7cd10e4501913369de756ed469 (127.0.97.2:46399)
I20260929 20:53:21.451196 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:53:21.466825 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32991
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:21.769196 4297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:21.769467 4297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:21.769605 4297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:21.776618 4297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:21.776724 4297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:21.776890 4297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:53:21.786047 4297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:21.787218 4297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:21.788450 4297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:21.800172 4302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:21.800235 4305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:21.800189 4303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:21.800884 4297 server_base.cc:1034] running on GCE node
I20260929 20:53:21.801479 4297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:21.802408 4297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:21.803633 4297 hybrid_clock.cc:648] HybridClock initialized: now 1790715201803585 us; error 42 us; skew 500 ppm
I20260929 20:53:21.803936 4297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:21 dist-test-slave-m9g1 krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:21.806803 4297 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:53:21.808081 4297 webserver.cc:460] Webserver started at http://127.0.97.3:44341/ using document root <none> and password file <none>
I20260929 20:53:21.808687 4297 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:21.808779 4297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:21.809013 4297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:21.811837 4297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "968814e7ca6c4b379c072a9b170e95b1"
format_stamp: "Formatted at 2026-09-29 20:53:21 on dist-test-slave-m9g1"
server_key: "6ffeba50adeb98999221471458271964"
server_key_iv: "50abf992bd147e571179bd756b8fe479"
server_key_version: "encryptionkey@0"
I20260929 20:53:21.812546 4297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "968814e7ca6c4b379c072a9b170e95b1"
format_stamp: "Formatted at 2026-09-29 20:53:21 on dist-test-slave-m9g1"
server_key: "6ffeba50adeb98999221471458271964"
server_key_iv: "50abf992bd147e571179bd756b8fe479"
server_key_version: "encryptionkey@0"
I20260929 20:53:21.817250 4297 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 20:53:21.820668 4312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:21.821933 4297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:53:21.822064 4297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "968814e7ca6c4b379c072a9b170e95b1"
format_stamp: "Formatted at 2026-09-29 20:53:21 on dist-test-slave-m9g1"
server_key: "6ffeba50adeb98999221471458271964"
server_key_iv: "50abf992bd147e571179bd756b8fe479"
server_key_version: "encryptionkey@0"
I20260929 20:53:21.822269 4297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:21.835718 4297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:21.839262 4297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:21.839557 4297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:21.840871 4297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:21.842356 4297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:21.842430 4297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:21.842543 4297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:21.842595 4297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:21.870889 4297 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35069
I20260929 20:53:21.870930 4424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35069 every 8 connection(s)
I20260929 20:53:21.872076 4297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:53:21.880812 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 4297
I20260929 20:53:21.880963 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:53:21.881453 388 external_mini_cluster.cc:1428] Setting key 45d4907a87c1b2b3b80b6d3e720d334e
Sep 29 20:53:21 dist-test-slave-m9g1 krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:21.890028 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:21.874385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55605 (local address 127.0.97.62:32991)
0929 20:53:21.874566 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:21.874572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:21.875734 (+ 1162us) server_negotiation.cc:407] Connection header received
0929 20:53:21.877419 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:21.877424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:21.877489 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:21.877572 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:21.878971 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:21.879671 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:21.881224 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:21.881803 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:21.881941 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:21.885156 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:21.885161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:21.885164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:21.885195 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:21.887515 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:21.888093 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:21.888097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:21.888099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:21.888202 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:21.888593 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:21.888596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:21.888598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:21.888837 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:21.888945 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:21.889630 (+ 685us) server_negotiation.cc:299] Negotiation successful
0929 20:53:21.889787 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:53:21.890942 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:21.874911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32991 (local address 127.0.97.3:55605)
0929 20:53:21.875589 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:21.875631 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:21.877177 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:21.877753 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:21.877760 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:21.878307 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:21.878847 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:21.878863 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:21.879800 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:21.879803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:21.881081 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:21.881092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:21.882031 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:21.882039 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:21.882321 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:21.882919 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:21.882940 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:21.884984 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:21.887629 (+ 2645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:21.887635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:21.887647 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:21.888007 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:21.888315 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:21.888319 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:21.888322 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:21.888505 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:21.888950 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:21.888956 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:21.889310 (+ 354us) client_negotiation.cc:769] Sending connection context
0929 20:53:21.889523 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 20:53:21.889890 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":280,"threads_started":1}
I20260929 20:53:21.892094 4425 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32991
I20260929 20:53:21.892387 4425 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:21.893040 4425 heartbeater.cc:511] Master 127.0.97.62:32991 requested a full tablet report, sending...
I20260929 20:53:21.894373 3969 ts_manager.cc:194] Registered new tserver with Master: 968814e7ca6c4b379c072a9b170e95b1 (127.0.97.3:35069)
I20260929 20:53:21.895407 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55605
I20260929 20:53:21.896437 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:53:21 dist-test-slave-m9g1 krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:53:21 dist-test-slave-m9g1 krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:53:21.938287 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:21.924296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38800 (local address 127.0.97.1:40163)
0929 20:53:21.924857 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:21.924867 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:21.924899 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:53:21.925083 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:21.925091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:21.925342 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:21.925486 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:21.926800 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:21.927891 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:21.929182 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:21.929707 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:21.929820 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:21.933392 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:21.933404 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:21.933418 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:21.933457 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:21.935968 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:21.936523 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:21.936530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:21.936532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:21.936647 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:21.936971 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:21.936975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:21.936977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:21.937308 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:21.937534 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:21.937720 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 20:53:21.937907 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":263,"threads_started":1}
Sep 29 20:53:21 dist-test-slave-m9g1 krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:21.955013 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:21.942336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.0.97.62:32991)
0929 20:53:21.942527 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:21.942533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:21.942699 (+ 166us) server_negotiation.cc:407] Connection header received
0929 20:53:21.942860 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:21.942865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:21.942923 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:21.943001 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:21.944130 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:21.944838 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:21.946101 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:21.946558 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:21.946665 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:21.950267 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:21.950272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:21.950276 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:21.950308 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:21.952647 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:21.953241 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:21.953246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:21.953249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:21.953370 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:21.953682 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:21.953687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:21.953690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:21.953989 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:21.954231 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:21.954516 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 20:53:21.954715 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 20:53:21.955570 3969 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38940
I20260929 20:53:21.956489 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 4030
I20260929 20:53:21.968221 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:21.957253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.0.97.1:40163)
0929 20:53:21.957446 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:21.957451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:21.957664 (+ 213us) server_negotiation.cc:407] Connection header received
0929 20:53:21.957811 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:21.957816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:21.957885 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:21.957962 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:21.959100 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:21.959764 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:21.960898 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:21.961298 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:21.961401 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:21.963441 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:21.963446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:21.963449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:21.963478 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:21.965787 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:21.966411 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:21.966415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:21.966418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:21.966527 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:21.966849 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:21.966852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:21.966854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:21.967113 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:21.967331 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:21.967687 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 20:53:21.967902 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:53:21.968784 4153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:22.049196 4158 heartbeater.cc:503] Master 127.0.97.62:32991 was elected leader, sending a full tablet report...
I20260929 20:53:22.049860 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 4030
W20260929 20:53:22.063048 3953 connection.cc:535] server connection from 127.0.97.1:48487 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 20:53:22.063285 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 4164
Sep 29 20:53:22 dist-test-slave-m9g1 krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:53:22.078063 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:22.064126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41054 (local address 127.0.97.2:46399)
0929 20:53:22.064602 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:22.064609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:22.064634 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:53:22.064697 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:22.064703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:22.064928 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:22.065055 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:22.066096 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:22.067115 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:22.068310 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:22.068846 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:22.068942 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:22.072596 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:22.072607 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:22.072622 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:22.072652 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:22.075306 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:22.076048 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:22.076056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:22.076058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:22.076170 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:22.076557 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:22.076561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:22.076565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:22.076989 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:22.077305 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:22.077497 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 20:53:22.077718 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":239,"threads_started":1}
I20260929 20:53:22.079346 4287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:22.159020 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 4164
I20260929 20:53:22.170960 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 4297
Sep 29 20:53:22 dist-test-slave-m9g1 krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:53:22.185225 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:22.171814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48812 (local address 127.0.97.3:35069)
0929 20:53:22.172326 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:22.172333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:22.172361 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:53:22.172417 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:22.172424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:22.172665 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:22.172811 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:22.173878 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:22.174933 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:22.176079 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:22.176592 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:22.176679 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:22.180375 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:22.180386 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:22.180400 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:22.180430 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:22.183016 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:22.183605 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:22.183610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:22.183613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:22.183716 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:22.183985 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:22.183988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:22.183990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:22.184301 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:22.184524 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:22.184697 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 20:53:22.184890 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":240,"threads_started":1}
I20260929 20:53:22.186625 4420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:22.266857 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 4297
I20260929 20:53:22.278944 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 3937
I20260929 20:53:22.280014 3999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:22.379299 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 3937
2026-09-29T20:53:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3565 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:53:22 dist-test-slave-m9g1 krb5kdc[4444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:53:22 dist-test-slave-m9g1 krb5kdc[4444](info): set up 2 sockets
Sep 29 20:53:22 dist-test-slave-m9g1 krb5kdc[4444](info): commencing operation
krb5kdc: starting...
W20260929 20:53:23.366252 388 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.007s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:53:23 dist-test-slave-m9g1 krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:53:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:53:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:53:23.533031 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43523
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:23.838018 4457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:23.838330 4457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:23.838451 4457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:23.845528 4457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:53:23.845633 4457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:23.845705 4457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:23.845839 4457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:53:23.845909 4457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:53:23.856519 4457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:23.857730 4457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:23.859052 4457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:23.869432 4462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:23.869460 4463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:23.869863 4465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:23.870172 4457 server_base.cc:1034] running on GCE node
I20260929 20:53:23.870875 4457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:23.872131 4457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:23.873314 4457 hybrid_clock.cc:648] HybridClock initialized: now 1790715203873262 us; error 49 us; skew 500 ppm
I20260929 20:53:23.873628 4457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:23 dist-test-slave-m9g1 krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:23.876598 4457 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:53:23.877668 4457 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:23.877753 4457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:23.878036 4457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:23.880846 4457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "2f4a978f715a4e12a6f5d9219ce62824"
format_stamp: "Formatted at 2026-09-29 20:53:23 on dist-test-slave-m9g1"
server_key: "5ce945785f2fdf9e47f0dcbe1a09a555"
server_key_iv: "30366bad9a56ab60ca2fa9742314b6c6"
server_key_version: "encryptionkey@0"
I20260929 20:53:23.881583 4457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "2f4a978f715a4e12a6f5d9219ce62824"
format_stamp: "Formatted at 2026-09-29 20:53:23 on dist-test-slave-m9g1"
server_key: "5ce945785f2fdf9e47f0dcbe1a09a555"
server_key_iv: "30366bad9a56ab60ca2fa9742314b6c6"
server_key_version: "encryptionkey@0"
I20260929 20:53:23.886428 4457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:53:23.889593 4471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:23.890851 4457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:53:23.890995 4457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "2f4a978f715a4e12a6f5d9219ce62824"
format_stamp: "Formatted at 2026-09-29 20:53:23 on dist-test-slave-m9g1"
server_key: "5ce945785f2fdf9e47f0dcbe1a09a555"
server_key_iv: "30366bad9a56ab60ca2fa9742314b6c6"
server_key_version: "encryptionkey@0"
I20260929 20:53:23.891180 4457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:23.916199 4457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:23.920009 4457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:23.920241 4457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:23.936028 4457 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43523
I20260929 20:53:23.936035 4522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43523 every 8 connection(s)
I20260929 20:53:23.936879 4457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:53:23.938040 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 4457
I20260929 20:53:23.938275 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:53:23.939023 388 external_mini_cluster.cc:1428] Setting key 76c36f527505f5b46ddaf69430238f7f
I20260929 20:53:23.941447 4523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:23.951531 4523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824: Bootstrap starting.
Sep 29 20:53:23 dist-test-slave-m9g1 krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:23.954737 4523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:23.955727 4523 log.cc:824] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:23.958235 4523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824: No bootstrap required, opened a new log
I20260929 20:53:23.959352 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:23.941049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.0.97.62:43523)
0929 20:53:23.941808 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:23.941820 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:23.941858 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:53:23.943480 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:23.943501 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:23.943893 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:23.944243 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:23.945640 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:23.946959 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:23.948135 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:23.948804 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:23.948979 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:23.952829 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:23.952837 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:23.952851 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:23.952890 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:23.955656 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:23.956296 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:23.956302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:23.956308 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:23.956471 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:23.956814 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:23.956817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:23.956819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:23.957308 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:23.957550 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:23.957972 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 20:53:23.958400 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":331,"threads_started":1}
I20260929 20:53:23.965314 4523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f4a978f715a4e12a6f5d9219ce62824" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43523 } }
I20260929 20:53:23.965620 4523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:23.965711 4523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4a978f715a4e12a6f5d9219ce62824, State: Initialized, Role: FOLLOWER
I20260929 20:53:23.966293 4523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f4a978f715a4e12a6f5d9219ce62824" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43523 } }
I20260929 20:53:23.966557 4523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:53:23.966699 4523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:53:23.966864 4523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:23.968837 4523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f4a978f715a4e12a6f5d9219ce62824" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43523 } }
I20260929 20:53:23.969269 4523 leader_election.cc:302] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f4a978f715a4e12a6f5d9219ce62824; no voters:
I20260929 20:53:23.969771 4523 leader_election.cc:288] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:53:23.969879 4528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:23.970490 4528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824 [term 1 LEADER]: Becoming Leader. State: Replica: 2f4a978f715a4e12a6f5d9219ce62824, State: Running, Role: LEADER
I20260929 20:53:23.970960 4528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f4a978f715a4e12a6f5d9219ce62824" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43523 } }
I20260929 20:53:23.971655 4523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:53:23.973487 4529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f4a978f715a4e12a6f5d9219ce62824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f4a978f715a4e12a6f5d9219ce62824" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43523 } } }
I20260929 20:53:23.973800 4529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:23.977432 4535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:53:23.977322 4530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f4a978f715a4e12a6f5d9219ce62824. Latest consensus state: current_term: 1 leader_uuid: "2f4a978f715a4e12a6f5d9219ce62824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f4a978f715a4e12a6f5d9219ce62824" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43523 } } }
I20260929 20:53:23.977545 4530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:23.982028 4535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:53:23.990235 4535 catalog_manager.cc:1347] Generated new cluster ID: e2fb68268d7047d8ad9391b8ac261e54
I20260929 20:53:23.990343 4535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:53:24.024528 4535 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:53:24.026401 4535 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:53:24.037952 4535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824: Generated new TSK 0
I20260929 20:53:24.039002 4535 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:53:24.097034 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43523
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:24.401275 4549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:24.401549 4549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:24.401651 4549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:24.408871 4549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:24.408977 4549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:24.409107 4549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:53:24.418613 4549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:24.419826 4549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:24.421025 4549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:24.432911 4554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:24.432941 4557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:24.432878 4555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:24.433737 4549 server_base.cc:1034] running on GCE node
I20260929 20:53:24.434350 4549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:24.435191 4549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:24.436380 4549 hybrid_clock.cc:648] HybridClock initialized: now 1790715204436335 us; error 41 us; skew 500 ppm
I20260929 20:53:24.436666 4549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:24 dist-test-slave-m9g1 krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:24.439476 4549 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:53:24.440517 4549 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:24.440620 4549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:24.440903 4549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:24.443730 4549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "751f34a565c84f50b583d113fc8e456b"
format_stamp: "Formatted at 2026-09-29 20:53:24 on dist-test-slave-m9g1"
server_key: "6d57a3202f8ac8b7b5ab5e0de39fb4df"
server_key_iv: "bc94dc62152192e008cd1950103efd48"
server_key_version: "encryptionkey@0"
I20260929 20:53:24.444499 4549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "751f34a565c84f50b583d113fc8e456b"
format_stamp: "Formatted at 2026-09-29 20:53:24 on dist-test-slave-m9g1"
server_key: "6d57a3202f8ac8b7b5ab5e0de39fb4df"
server_key_iv: "bc94dc62152192e008cd1950103efd48"
server_key_version: "encryptionkey@0"
I20260929 20:53:24.449090 4549 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:53:24.452256 4563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:24.453492 4549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:53:24.453644 4549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "751f34a565c84f50b583d113fc8e456b"
format_stamp: "Formatted at 2026-09-29 20:53:24 on dist-test-slave-m9g1"
server_key: "6d57a3202f8ac8b7b5ab5e0de39fb4df"
server_key_iv: "bc94dc62152192e008cd1950103efd48"
server_key_version: "encryptionkey@0"
I20260929 20:53:24.453822 4549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:24.464283 4549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:24.467790 4549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:24.468009 4549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:24.468838 4549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:24.469985 4549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:24.470084 4549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:24.470239 4549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:24.470311 4549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 20:53:24.498894 4549 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44091
I20260929 20:53:24.498955 4675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44091 every 8 connection(s)
I20260929 20:53:24.499677 4549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:53:24.502353 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 4549
I20260929 20:53:24.502540 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:53:24.503274 388 external_mini_cluster.cc:1428] Setting key 477d890a05a0e29d9f817427c9b59ef5
Sep 29 20:53:24 dist-test-slave-m9g1 krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:24.519341 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:24.501864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38103 (local address 127.0.97.62:43523)
0929 20:53:24.502374 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:24.502380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:24.503875 (+ 1495us) server_negotiation.cc:407] Connection header received
0929 20:53:24.505484 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:24.505488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:24.505550 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:24.505635 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:24.507464 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:24.508154 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:24.510126 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:24.510529 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:24.510632 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:24.513921 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:24.513926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:24.513929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:24.513958 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:24.516308 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:24.517116 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:24.517119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:24.517121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:24.517252 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:24.517615 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:24.517619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:24.517623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:24.517985 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:24.518129 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:24.518752 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 20:53:24.518980 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":233,"threads_started":1}
I20260929 20:53:24.520005 4679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:24.502444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43523 (local address 127.0.97.1:38103)
0929 20:53:24.503242 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:24.503301 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:24.505209 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:24.505874 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:24.505885 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:24.506575 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:24.507315 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:24.507339 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:24.508284 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:24.508289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:24.510018 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:24.510031 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:24.510732 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:24.510737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:24.510924 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:24.511515 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:24.511538 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:24.513708 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:24.516437 (+ 2729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:24.516446 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:24.516461 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:24.517018 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:24.517354 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:24.517357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:24.517359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:24.517522 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:24.518097 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:24.518103 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:24.518423 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 20:53:24.518638 (+ 215us) client_negotiation.cc:240] Negotiation successful
0929 20:53:24.518984 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":294,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:53:24.521449 4676 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43523
I20260929 20:53:24.521713 4676 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:24.522374 4676 heartbeater.cc:511] Master 127.0.97.62:43523 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 20:53:24.524435 4488 ts_manager.cc:194] Registered new tserver with Master: 751f34a565c84f50b583d113fc8e456b (127.0.97.1:44091)
I20260929 20:53:24.526292 4488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:53:24.544507 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43523
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:24.846331 4682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:24.846617 4682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:24.846755 4682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:24.854704 4682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:24.854815 4682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:24.854987 4682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:53:24.864516 4682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:24.865712 4682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:24.866977 4682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:24.878394 4687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:24.878479 4688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:24.878551 4690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:24.879317 4682 server_base.cc:1034] running on GCE node
I20260929 20:53:24.879871 4682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:24.880666 4682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:24.881841 4682 hybrid_clock.cc:648] HybridClock initialized: now 1790715204881796 us; error 39 us; skew 500 ppm
I20260929 20:53:24.882108 4682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:24 dist-test-slave-m9g1 krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:24.884948 4682 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:53:24.885968 4682 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:24.886065 4682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:24.886382 4682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:24.889050 4682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "de968ae79263448eb14d9a43637ebff1"
format_stamp: "Formatted at 2026-09-29 20:53:24 on dist-test-slave-m9g1"
server_key: "a9c1fd65f431196a0b6865738a182d78"
server_key_iv: "2857e75800ebe13a840df4c9439bc465"
server_key_version: "encryptionkey@0"
I20260929 20:53:24.889791 4682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "de968ae79263448eb14d9a43637ebff1"
format_stamp: "Formatted at 2026-09-29 20:53:24 on dist-test-slave-m9g1"
server_key: "a9c1fd65f431196a0b6865738a182d78"
server_key_iv: "2857e75800ebe13a840df4c9439bc465"
server_key_version: "encryptionkey@0"
I20260929 20:53:24.894291 4682 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260929 20:53:24.897385 4696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:24.898589 4682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 20:53:24.898730 4682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "de968ae79263448eb14d9a43637ebff1"
format_stamp: "Formatted at 2026-09-29 20:53:24 on dist-test-slave-m9g1"
server_key: "a9c1fd65f431196a0b6865738a182d78"
server_key_iv: "2857e75800ebe13a840df4c9439bc465"
server_key_version: "encryptionkey@0"
I20260929 20:53:24.898918 4682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:24.911737 4682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:24.915208 4682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:24.915444 4682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:24.916260 4682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:24.917430 4682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:24.917512 4682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:24.917629 4682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:24.917702 4682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:24.946245 4682 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35661
I20260929 20:53:24.946292 4808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35661 every 8 connection(s)
I20260929 20:53:24.947031 4682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:53:24.948277 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 4682
I20260929 20:53:24.948464 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:53:24.949025 388 external_mini_cluster.cc:1428] Setting key 83ebd74fde1b334021424f59a0320752
Sep 29 20:53:24 dist-test-slave-m9g1 krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:53:24.966766 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:24.949859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53399 (local address 127.0.97.62:43523)
0929 20:53:24.950033 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:24.950039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:24.952148 (+ 2109us) server_negotiation.cc:407] Connection header received
0929 20:53:24.953826 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:24.953831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:24.953911 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:24.953992 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:24.955458 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:24.956155 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:24.957756 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:24.958226 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:24.958335 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:24.961653 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:24.961658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:24.961661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:24.961690 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:24.963985 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:24.964623 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:24.964629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:24.964632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:24.964754 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:24.965156 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:24.965161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:24.965165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:24.965404 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:24.965537 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:24.966244 (+ 707us) server_negotiation.cc:299] Negotiation successful
0929 20:53:24.966451 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 20:53:24.967748 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:24.950446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43523 (local address 127.0.97.2:53399)
0929 20:53:24.952004 (+ 1558us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:24.952043 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:24.953578 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:24.954265 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:24.954273 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:24.954761 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:24.955301 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:24.955318 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:24.956302 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:24.956307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:24.957626 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:24.957641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:24.958444 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:24.958451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:24.958654 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:24.959261 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:24.959298 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:24.961455 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:24.964106 (+ 2651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:24.964115 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:24.964130 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:24.964517 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:24.964858 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:24.964862 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:24.964866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:24.965059 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:24.965521 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:24.965528 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:24.965849 (+ 321us) client_negotiation.cc:769] Sending connection context
0929 20:53:24.966129 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 20:53:24.966631 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1339,"thread_start_us":382,"threads_started":1}
I20260929 20:53:24.969184 4809 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43523
I20260929 20:53:24.969466 4809 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:24.970054 4809 heartbeater.cc:511] Master 127.0.97.62:43523 requested a full tablet report, sending...
I20260929 20:53:24.971400 4488 ts_manager.cc:194] Registered new tserver with Master: de968ae79263448eb14d9a43637ebff1 (127.0.97.2:35661)
I20260929 20:53:24.972165 4488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:53:24.987293 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43523
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:25.289143 4814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:25.289398 4814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:25.289510 4814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:25.296511 4814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:25.296618 4814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:25.296737 4814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:53:25.306700 4814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:25.307874 4814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:25.309064 4814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:25.320310 4819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:25.320379 4822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:25.320398 4820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:25.321234 4814 server_base.cc:1034] running on GCE node
I20260929 20:53:25.321738 4814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:25.322571 4814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:25.323765 4814 hybrid_clock.cc:648] HybridClock initialized: now 1790715205323725 us; error 27 us; skew 500 ppm
I20260929 20:53:25.324041 4814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:25 dist-test-slave-m9g1 krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:25.326933 4814 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:53:25.327919 4814 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:25.328015 4814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:25.328315 4814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:25.331143 4814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "fadb2c302aeb4b9c87e5f0c652f340d3"
format_stamp: "Formatted at 2026-09-29 20:53:25 on dist-test-slave-m9g1"
server_key: "993ae36e9b51d848eb040380118c5613"
server_key_iv: "ca1f8ad5b761e5e40024aa3088b21d6b"
server_key_version: "encryptionkey@0"
I20260929 20:53:25.331899 4814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fadb2c302aeb4b9c87e5f0c652f340d3"
format_stamp: "Formatted at 2026-09-29 20:53:25 on dist-test-slave-m9g1"
server_key: "993ae36e9b51d848eb040380118c5613"
server_key_iv: "ca1f8ad5b761e5e40024aa3088b21d6b"
server_key_version: "encryptionkey@0"
I20260929 20:53:25.336606 4814 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:53:25.339802 4828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:25.341012 4814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 20:53:25.341157 4814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "fadb2c302aeb4b9c87e5f0c652f340d3"
format_stamp: "Formatted at 2026-09-29 20:53:25 on dist-test-slave-m9g1"
server_key: "993ae36e9b51d848eb040380118c5613"
server_key_iv: "ca1f8ad5b761e5e40024aa3088b21d6b"
server_key_version: "encryptionkey@0"
I20260929 20:53:25.341341 4814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:25.362803 4814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:25.366508 4814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:25.366751 4814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:25.367664 4814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:25.368916 4814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:25.369014 4814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:25.369123 4814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:25.369192 4814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:25.397472 4814 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35243
I20260929 20:53:25.397521 4940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35243 every 8 connection(s)
I20260929 20:53:25.398277 4814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:53:25.403508 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 4814
I20260929 20:53:25.403692 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:53:25.404181 388 external_mini_cluster.cc:1428] Setting key b310c944b17bf262c12e29aa3ba67c39
Sep 29 20:53:25 dist-test-slave-m9g1 krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:25.428154 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:25.400452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59897 (local address 127.0.97.62:43523)
0929 20:53:25.400610 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:25.400614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:25.401784 (+ 1170us) server_negotiation.cc:407] Connection header received
0929 20:53:25.403804 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:25.403809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:25.403872 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:25.403953 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:25.405709 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.406771 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.408294 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.408690 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.408778 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:25.422946 (+ 14168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:25.422951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:25.422955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:25.422988 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:25.425293 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:25.426123 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:25.426127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:25.426129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:25.426287 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:25.426731 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:25.426734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:25.426736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:25.426992 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:25.427103 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:25.427757 (+ 654us) server_negotiation.cc:299] Negotiation successful
0929 20:53:25.427921 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:53:25.429250 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:25.400910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43523 (local address 127.0.97.3:59897)
0929 20:53:25.401632 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:25.401681 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:25.403516 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:25.404191 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:25.404203 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:25.404822 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:25.405550 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:25.405577 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.406920 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.406925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:25.408182 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:25.408194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.408900 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.408905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:25.409130 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:25.420392 (+ 11262us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:25.420420 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:25.422728 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:25.425448 (+ 2720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:25.425456 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:25.425472 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:25.426013 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:25.426421 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:25.426425 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:25.426428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:25.426638 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:25.427089 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:25.427094 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:25.427449 (+ 355us) client_negotiation.cc:769] Sending connection context
0929 20:53:25.427692 (+ 243us) client_negotiation.cc:240] Negotiation successful
0929 20:53:25.428090 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":286,"threads_started":1}
I20260929 20:53:25.430611 4941 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43523
I20260929 20:53:25.430918 4941 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:25.431578 4941 heartbeater.cc:511] Master 127.0.97.62:43523 requested a full tablet report, sending...
I20260929 20:53:25.432869 4488 ts_manager.cc:194] Registered new tserver with Master: fadb2c302aeb4b9c87e5f0c652f340d3 (127.0.97.3:35243)
I20260929 20:53:25.433614 4488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59897
I20260929 20:53:25.443233 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:53:25.458951 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:25.447493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50678 (local address 127.0.97.62:43523)
0929 20:53:25.447668 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:25.447674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:25.448034 (+ 360us) server_negotiation.cc:407] Connection header received
0929 20:53:25.448187 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:25.448192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:25.448260 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:25.448336 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:25.449661 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.450431 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.451600 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.452042 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.452146 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:25.454227 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:25.454234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:25.454239 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:25.454273 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:25.456556 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:25.457144 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:25.457148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:25.457150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:25.457261 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:25.457594 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:25.457599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:25.457603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:25.457939 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:25.458144 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:25.458476 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 20:53:25.458676 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 20:53:25.462602 4488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:53:25.469168 4488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:53:25.505133 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:25.480831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35661 (local address 127.0.0.1:38246)
0929 20:53:25.495292 (+ 14461us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:25.495311 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:25.495459 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:25.496636 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:25.496644 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:25.496681 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:25.497038 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:25.497049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.498730 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.498735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:25.501018 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:25.501031 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.504249 (+ 3218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.504255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:25.504442 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:25.504490 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 20:53:25.504622 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 20:53:25.504789 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14325,"thread_start_us":12539,"threads_started":1}
I20260929 20:53:25.505849 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:25.480956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.0.97.2:35661)
0929 20:53:25.494485 (+ 13529us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:25.494495 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:25.495445 (+ 950us) server_negotiation.cc:407] Connection header received
0929 20:53:25.495842 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:25.495852 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:25.496134 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:25.496312 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:25.497190 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.498579 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.501207 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.503834 (+ 2627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.503953 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:25.504130 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:25.505369 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0929 20:53:25.505545 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13347,"thread_start_us":13573,"threads_started":1}
I20260929 20:53:25.506253 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:25.480089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44091 (local address 127.0.0.1:47848)
0929 20:53:25.493733 (+ 13644us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:25.493776 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:25.493927 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:25.498582 (+ 4655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:25.498587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:25.498612 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:25.499042 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:25.499053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.501051 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.501056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:25.503371 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:25.503385 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.505361 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.505367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:25.505586 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:25.505616 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:53:25.505695 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 20:53:25.505812 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13407,"mutex_wait_us":68,"thread_start_us":15454,"threads_started":1}
I20260929 20:53:25.506570 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:25.480319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47848 (local address 127.0.97.1:44091)
0929 20:53:25.497874 (+ 17555us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:25.497881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:25.497903 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:53:25.497982 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:25.497989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:25.498208 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:25.498359 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:25.499216 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.500814 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.503536 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.504918 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.505026 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:25.505227 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:25.506104 (+ 877us) server_negotiation.cc:299] Negotiation successful
0929 20:53:25.506276 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17410,"thread_start_us":17290,"threads_started":1}
I20260929 20:53:25.506935 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:25.481243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58218 (local address 127.0.97.3:35243)
0929 20:53:25.497609 (+ 16366us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:25.497617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:25.497649 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:53:25.497737 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:25.497744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:25.497981 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:25.498873 (+ 892us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:25.499155 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.500655 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.503449 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.504767 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.504876 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:25.505089 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:25.506475 (+ 1386us) server_negotiation.cc:299] Negotiation successful
0929 20:53:25.506630 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16214,"thread_start_us":16068,"threads_started":1}
I20260929 20:53:25.506351 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:25.480583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35243 (local address 127.0.0.1:58218)
0929 20:53:25.495122 (+ 14539us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:25.495141 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:25.495258 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:25.498168 (+ 2910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:25.498173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:25.498221 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:25.498690 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:25.498703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.500803 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.500808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:25.503270 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:25.503285 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.505227 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.505233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:25.505719 (+ 486us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:25.505750 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:53:25.505824 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 20:53:25.505946 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14422,"mutex_wait_us":87,"thread_start_us":14273,"threads_started":1}
I20260929 20:53:25.509877 4611 tablet_service.cc:1467] Processing CreateTablet for tablet b000450817d0461a95932d50fd7411d5 (DEFAULT_TABLE table=test-table [id=99d8ffa8f1a7404eb7c6c62e6531131b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:25.509761 4744 tablet_service.cc:1467] Processing CreateTablet for tablet b000450817d0461a95932d50fd7411d5 (DEFAULT_TABLE table=test-table [id=99d8ffa8f1a7404eb7c6c62e6531131b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:25.510363 4876 tablet_service.cc:1467] Processing CreateTablet for tablet b000450817d0461a95932d50fd7411d5 (DEFAULT_TABLE table=test-table [id=99d8ffa8f1a7404eb7c6c62e6531131b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:25.511065 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b000450817d0461a95932d50fd7411d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:25.511117 4744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b000450817d0461a95932d50fd7411d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:25.511498 4876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b000450817d0461a95932d50fd7411d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:25.521622 4959 tablet_bootstrap.cc:492] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b: Bootstrap starting.
I20260929 20:53:25.524734 4959 tablet_bootstrap.cc:654] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:25.524771 4960 tablet_bootstrap.cc:492] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3: Bootstrap starting.
I20260929 20:53:25.526046 4959 log.cc:824] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:25.527426 4961 tablet_bootstrap.cc:492] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1: Bootstrap starting.
I20260929 20:53:25.527987 4960 tablet_bootstrap.cc:654] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:25.529145 4960 log.cc:824] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:25.529219 4959 tablet_bootstrap.cc:492] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b: No bootstrap required, opened a new log
I20260929 20:53:25.529294 4676 heartbeater.cc:503] Master 127.0.97.62:43523 was elected leader, sending a full tablet report...
I20260929 20:53:25.529542 4959 ts_tablet_manager.cc:1397] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 20:53:25.533998 4961 tablet_bootstrap.cc:654] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:25.535131 4961 log.cc:824] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:25.536830 4960 tablet_bootstrap.cc:492] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3: No bootstrap required, opened a new log
I20260929 20:53:25.537106 4960 ts_tablet_manager.cc:1397] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260929 20:53:25.537417 4959 raft_consensus.cc:348] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f34a565c84f50b583d113fc8e456b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44091 } } peers { permanent_uuid: "de968ae79263448eb14d9a43637ebff1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35661 } } peers { permanent_uuid: "fadb2c302aeb4b9c87e5f0c652f340d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35243 } }
I20260929 20:53:25.537724 4959 raft_consensus.cc:374] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:25.537845 4959 raft_consensus.cc:729] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751f34a565c84f50b583d113fc8e456b, State: Initialized, Role: FOLLOWER
I20260929 20:53:25.537914 4961 tablet_bootstrap.cc:492] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1: No bootstrap required, opened a new log
I20260929 20:53:25.538204 4961 ts_tablet_manager.cc:1397] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 20:53:25.538420 4959 consensus_queue.cc:260] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f34a565c84f50b583d113fc8e456b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44091 } } peers { permanent_uuid: "de968ae79263448eb14d9a43637ebff1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35661 } } peers { permanent_uuid: "fadb2c302aeb4b9c87e5f0c652f340d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35243 } }
I20260929 20:53:25.539942 4959 ts_tablet_manager.cc:1428] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260929 20:53:25.545470 4960 raft_consensus.cc:348] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f34a565c84f50b583d113fc8e456b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44091 } } peers { permanent_uuid: "de968ae79263448eb14d9a43637ebff1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35661 } } peers { permanent_uuid: "fadb2c302aeb4b9c87e5f0c652f340d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35243 } }
I20260929 20:53:25.545884 4960 raft_consensus.cc:374] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:25.545985 4960 raft_consensus.cc:729] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fadb2c302aeb4b9c87e5f0c652f340d3, State: Initialized, Role: FOLLOWER
I20260929 20:53:25.546588 4960 consensus_queue.cc:260] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f34a565c84f50b583d113fc8e456b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44091 } } peers { permanent_uuid: "de968ae79263448eb14d9a43637ebff1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35661 } } peers { permanent_uuid: "fadb2c302aeb4b9c87e5f0c652f340d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35243 } }
I20260929 20:53:25.547086 4961 raft_consensus.cc:348] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f34a565c84f50b583d113fc8e456b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44091 } } peers { permanent_uuid: "de968ae79263448eb14d9a43637ebff1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35661 } } peers { permanent_uuid: "fadb2c302aeb4b9c87e5f0c652f340d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35243 } }
I20260929 20:53:25.547398 4961 raft_consensus.cc:374] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:25.547492 4961 raft_consensus.cc:729] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de968ae79263448eb14d9a43637ebff1, State: Initialized, Role: FOLLOWER
I20260929 20:53:25.547593 4941 heartbeater.cc:503] Master 127.0.97.62:43523 was elected leader, sending a full tablet report...
I20260929 20:53:25.548147 4960 ts_tablet_manager.cc:1428] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260929 20:53:25.548060 4961 consensus_queue.cc:260] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f34a565c84f50b583d113fc8e456b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44091 } } peers { permanent_uuid: "de968ae79263448eb14d9a43637ebff1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35661 } } peers { permanent_uuid: "fadb2c302aeb4b9c87e5f0c652f340d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35243 } }
I20260929 20:53:25.549003 4809 heartbeater.cc:503] Master 127.0.97.62:43523 was elected leader, sending a full tablet report...
I20260929 20:53:25.549616 4961 ts_tablet_manager.cc:1428] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20260929 20:53:25.617903 4966 raft_consensus.cc:482] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:53:25.618165 4966 raft_consensus.cc:504] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f34a565c84f50b583d113fc8e456b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44091 } } peers { permanent_uuid: "de968ae79263448eb14d9a43637ebff1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35661 } } peers { permanent_uuid: "fadb2c302aeb4b9c87e5f0c652f340d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35243 } }
I20260929 20:53:25.619788 4966 leader_election.cc:288] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 751f34a565c84f50b583d113fc8e456b (127.0.97.1:44091), de968ae79263448eb14d9a43637ebff1 (127.0.97.2:35661)
I20260929 20:53:25.625012 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:25.620323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55603 (local address 127.0.97.2:35661)
0929 20:53:25.620553 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:25.620559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:25.620579 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:53:25.620643 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:25.620646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:25.620701 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:25.620772 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:25.621189 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.621898 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.623479 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.624212 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.624329 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:25.624415 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:25.624660 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 20:53:25.624775 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 20:53:25.625034 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:25.620047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35661 (local address 127.0.97.3:55603)
0929 20:53:25.620349 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:25.620368 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:25.620495 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:25.620813 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:25.620818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:25.620844 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:25.621052 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:25.621063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.622020 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.622026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:25.623327 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:25.623342 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.624354 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.624359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:25.624519 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:25.624543 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:53:25.624619 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 20:53:25.624740 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":1}
I20260929 20:53:25.625655 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:25.620354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60135 (local address 127.0.97.1:44091)
0929 20:53:25.620524 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:25.620530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:25.620905 (+ 375us) server_negotiation.cc:407] Connection header received
0929 20:53:25.621098 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:25.621102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:25.621158 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:25.621229 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:25.621726 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.622463 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.624075 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.624799 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.624912 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:25.625002 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:25.625343 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 20:53:25.625463 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 20:53:25.625727 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:25.620132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44091 (local address 127.0.97.3:60135)
0929 20:53:25.620787 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:25.620810 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:25.620967 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:25.621261 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:25.621265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:25.621286 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:25.621575 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:25.621582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.622600 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.622603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:25.623935 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:25.623945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.624995 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.625001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:25.625159 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:25.625194 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 20:53:25.625275 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 20:53:25.625388 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":19,"spinlock_wait_cycles":28032,"thread_start_us":733,"threads_started":1}
I20260929 20:53:25.626019 4764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b000450817d0461a95932d50fd7411d5" candidate_uuid: "fadb2c302aeb4b9c87e5f0c652f340d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de968ae79263448eb14d9a43637ebff1" is_pre_election: true
I20260929 20:53:25.626564 4764 raft_consensus.cc:2454] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fadb2c302aeb4b9c87e5f0c652f340d3 in term 0.
I20260929 20:53:25.626724 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b000450817d0461a95932d50fd7411d5" candidate_uuid: "fadb2c302aeb4b9c87e5f0c652f340d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751f34a565c84f50b583d113fc8e456b" is_pre_election: true
I20260929 20:53:25.627157 4829 leader_election.cc:302] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de968ae79263448eb14d9a43637ebff1, fadb2c302aeb4b9c87e5f0c652f340d3; no voters:
I20260929 20:53:25.627223 4631 raft_consensus.cc:2454] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fadb2c302aeb4b9c87e5f0c652f340d3 in term 0.
I20260929 20:53:25.627496 4966 raft_consensus.cc:2781] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:53:25.627645 4966 raft_consensus.cc:482] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:53:25.627748 4966 raft_consensus.cc:3037] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:25.629822 4966 raft_consensus.cc:504] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f34a565c84f50b583d113fc8e456b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44091 } } peers { permanent_uuid: "de968ae79263448eb14d9a43637ebff1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35661 } } peers { permanent_uuid: "fadb2c302aeb4b9c87e5f0c652f340d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35243 } }
I20260929 20:53:25.630466 4966 leader_election.cc:288] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [CANDIDATE]: Term 1 election: Requested vote from peers 751f34a565c84f50b583d113fc8e456b (127.0.97.1:44091), de968ae79263448eb14d9a43637ebff1 (127.0.97.2:35661)
I20260929 20:53:25.630757 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b000450817d0461a95932d50fd7411d5" candidate_uuid: "fadb2c302aeb4b9c87e5f0c652f340d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751f34a565c84f50b583d113fc8e456b"
I20260929 20:53:25.630915 4631 raft_consensus.cc:3037] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:25.630967 4764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b000450817d0461a95932d50fd7411d5" candidate_uuid: "fadb2c302aeb4b9c87e5f0c652f340d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de968ae79263448eb14d9a43637ebff1"
I20260929 20:53:25.631151 4764 raft_consensus.cc:3037] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:25.632954 4631 raft_consensus.cc:2454] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fadb2c302aeb4b9c87e5f0c652f340d3 in term 1.
I20260929 20:53:25.633165 4764 raft_consensus.cc:2454] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fadb2c302aeb4b9c87e5f0c652f340d3 in term 1.
I20260929 20:53:25.633262 4830 leader_election.cc:302] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 751f34a565c84f50b583d113fc8e456b, fadb2c302aeb4b9c87e5f0c652f340d3; no voters:
I20260929 20:53:25.633543 4966 raft_consensus.cc:2781] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:25.633924 4966 raft_consensus.cc:686] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [term 1 LEADER]: Becoming Leader. State: Replica: fadb2c302aeb4b9c87e5f0c652f340d3, State: Running, Role: LEADER
I20260929 20:53:25.634415 4966 consensus_queue.cc:237] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f34a565c84f50b583d113fc8e456b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44091 } } peers { permanent_uuid: "de968ae79263448eb14d9a43637ebff1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35661 } } peers { permanent_uuid: "fadb2c302aeb4b9c87e5f0c652f340d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35243 } }
I20260929 20:53:25.638757 4488 catalog_manager.cc:5495] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to fadb2c302aeb4b9c87e5f0c652f340d3 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fadb2c302aeb4b9c87e5f0c652f340d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f34a565c84f50b583d113fc8e456b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de968ae79263448eb14d9a43637ebff1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fadb2c302aeb4b9c87e5f0c652f340d3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35243 } health_report { overall_health: HEALTHY } } }
W20260929 20:53:25.649559 4942 tablet.cc:2369] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:53:25.677699 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:25.672860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58234 (local address 127.0.97.3:35243)
0929 20:53:25.673023 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:25.673028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:25.673155 (+ 127us) server_negotiation.cc:407] Connection header received
0929 20:53:25.673341 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:25.673345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:25.673419 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:25.673531 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:25.673804 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.674649 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.675971 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.676355 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.676445 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:25.676811 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:25.677288 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:25.677395 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:25.677453 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 20:53:25.677516 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 20:53:25.685420 4631 raft_consensus.cc:1261] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b [term 1 FOLLOWER]: Refusing update from remote peer fadb2c302aeb4b9c87e5f0c652f340d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:25.685420 4764 raft_consensus.cc:1261] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1 [term 1 FOLLOWER]: Refusing update from remote peer fadb2c302aeb4b9c87e5f0c652f340d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:25.686394 4969 consensus_queue.cc:1035] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de968ae79263448eb14d9a43637ebff1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:25.686715 4966 consensus_queue.cc:1035] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "751f34a565c84f50b583d113fc8e456b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:25.710071 4488 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:25.710325 4488 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:25.713420 4488 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f4a978f715a4e12a6f5d9219ce62824: Sending DeleteTablet for 3 replicas of tablet b000450817d0461a95932d50fd7411d5
I20260929 20:53:25.714517 4611 tablet_service.cc:1514] Processing DeleteTablet for tablet b000450817d0461a95932d50fd7411d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47848
I20260929 20:53:25.714650 4876 tablet_service.cc:1514] Processing DeleteTablet for tablet b000450817d0461a95932d50fd7411d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58218
I20260929 20:53:25.714694 4744 tablet_service.cc:1514] Processing DeleteTablet for tablet b000450817d0461a95932d50fd7411d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38246
I20260929 20:53:25.716711 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 4549
I20260929 20:53:25.727032 4982 tablet_replica.cc:330] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b: stopping tablet replica
I20260929 20:53:25.727483 4982 raft_consensus.cc:2229] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:25.727999 4982 raft_consensus.cc:2258] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:25.728775 4984 tablet_replica.cc:330] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1: stopping tablet replica
I20260929 20:53:25.729306 4984 raft_consensus.cc:2229] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:25.729817 4984 raft_consensus.cc:2258] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:25.732403 4982 ts_tablet_manager.cc:1905] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:25.732954 4983 tablet_replica.cc:330] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3: stopping tablet replica
I20260929 20:53:25.733377 4983 raft_consensus.cc:2229] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:53:25.734071 4983 raft_consensus.cc:2258] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:25.735637 4984 ts_tablet_manager.cc:1905] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:25.735800 4983 ts_tablet_manager.cc:1905] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:25.739605 4982 ts_tablet_manager.cc:1918] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:25.739854 4982 log.cc:1196] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/b000450817d0461a95932d50fd7411d5
I20260929 20:53:25.740365 4982 ts_tablet_manager.cc:1939] T b000450817d0461a95932d50fd7411d5 P 751f34a565c84f50b583d113fc8e456b: Deleting consensus metadata
Sep 29 20:53:25 dist-test-slave-m9g1 krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:53:25.743646 4984 ts_tablet_manager.cc:1918] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:25.743795 4473 catalog_manager.cc:4841] TS 751f34a565c84f50b583d113fc8e456b (127.0.97.1:44091): tablet b000450817d0461a95932d50fd7411d5 (table test-table [id=99d8ffa8f1a7404eb7c6c62e6531131b]) successfully deleted
I20260929 20:53:25.743943 4984 log.cc:1196] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/b000450817d0461a95932d50fd7411d5
I20260929 20:53:25.744428 4984 ts_tablet_manager.cc:1939] T b000450817d0461a95932d50fd7411d5 P de968ae79263448eb14d9a43637ebff1: Deleting consensus metadata
I20260929 20:53:25.744451 4983 ts_tablet_manager.cc:1918] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:25.744663 4983 log.cc:1196] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/b000450817d0461a95932d50fd7411d5
I20260929 20:53:25.745308 4983 ts_tablet_manager.cc:1939] T b000450817d0461a95932d50fd7411d5 P fadb2c302aeb4b9c87e5f0c652f340d3: Deleting consensus metadata
I20260929 20:53:25.746281 4472 catalog_manager.cc:4841] TS de968ae79263448eb14d9a43637ebff1 (127.0.97.2:35661): tablet b000450817d0461a95932d50fd7411d5 (table test-table [id=99d8ffa8f1a7404eb7c6c62e6531131b]) successfully deleted
I20260929 20:53:25.747059 4474 catalog_manager.cc:4841] TS fadb2c302aeb4b9c87e5f0c652f340d3 (127.0.97.3:35243): tablet b000450817d0461a95932d50fd7411d5 (table test-table [id=99d8ffa8f1a7404eb7c6c62e6531131b]) successfully deleted
I20260929 20:53:25.749666 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:25.718704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47856 (local address 127.0.97.1:44091)
0929 20:53:25.718893 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:25.718899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:25.732671 (+ 13772us) server_negotiation.cc:407] Connection header received
0929 20:53:25.733076 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:25.733081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:25.733159 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:25.733269 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:25.734943 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.736067 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.737782 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.738466 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.738556 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:25.743597 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:25.743610 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:25.743628 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:25.743668 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:25.747152 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:25.747776 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:25.747784 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:25.747786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:25.747904 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:25.748269 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:25.748273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:25.748275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:25.748617 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:25.748882 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:25.749165 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 20:53:25.749373 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:53:25.750293 4671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:25.850481 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 4549
I20260929 20:53:25.870533 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 4682
Sep 29 20:53:25 dist-test-slave-m9g1 krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:53:25.884527 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:25.871236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38252 (local address 127.0.97.2:35661)
0929 20:53:25.871400 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:25.871404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:25.871422 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:53:25.871606 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:25.871610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:25.871703 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:25.871848 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:25.873024 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.873726 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.875046 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.875417 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.875524 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:25.879234 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:25.879248 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:25.879263 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:25.879300 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:25.881908 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:25.882688 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:25.882697 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:25.882700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:25.882818 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:25.883163 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:25.883166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:25.883168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:25.883510 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:25.883775 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:25.884041 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 20:53:25.884233 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:53:25.885100 4804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:25.971642 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 4682
I20260929 20:53:25.984942 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 4814
Sep 29 20:53:25 dist-test-slave-m9g1 krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:53:26.001519 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:25.985630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.0.97.3:35243)
0929 20:53:25.985785 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:25.985789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:25.985852 (+ 63us) server_negotiation.cc:407] Connection header received
0929 20:53:25.986047 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:25.986051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:25.986111 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:25.986214 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:25.987858 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.988695 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.990465 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:25.990893 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:25.990986 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:25.995881 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:25.995893 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:25.995908 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:25.995948 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:25.998915 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:25.999531 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:25.999539 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:25.999541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:25.999688 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:26.000036 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:26.000042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:26.000045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:26.000408 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:26.000663 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:26.000988 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 20:53:26.001201 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:53:26.002120 4936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:26.118978 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 4814
I20260929 20:53:26.133843 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 4457
I20260929 20:53:26.134908 4518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:26.238422 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 4457
2026-09-29T20:53:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3859 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T20:53:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:53:26Z Disabled control of system clock
I20260929 20:53:26.296247 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38957
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 20:53:26.607869 4993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:26.608124 4993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:26.608209 4993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:26.615216 4993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:53:26.615320 4993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:26.615365 4993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:26.615420 4993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:53:26.615463 4993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:53:26.624388 4993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:26.625607 4993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:26.627282 4993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:26.638311 4998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:26.638334 4999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:26.638418 5001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:26.643479 4993 server_base.cc:1034] running on GCE node
I20260929 20:53:26.644258 4993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:26.645704 4993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:26.646916 4993 hybrid_clock.cc:648] HybridClock initialized: now 1790715206646869 us; error 39 us; skew 500 ppm
I20260929 20:53:26.647274 4993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:53:26.649430 4993 webserver.cc:460] Webserver started at http://127.0.97.62:32997/ using document root <none> and password file <none>
I20260929 20:53:26.650091 4993 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:26.650203 4993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:26.650539 4993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:26.653326 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "1844967032464e31bf72323ba5c4d7b0"
format_stamp: "Formatted at 2026-09-29 20:53:26 on dist-test-slave-m9g1"
server_key: "c5d7f50bf3a5082caf22c7482ca82c4a"
server_key_iv: "8846eebb8b77da9055e2b3f07d44ddff"
server_key_version: "encryptionkey@0"
I20260929 20:53:26.654064 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "1844967032464e31bf72323ba5c4d7b0"
format_stamp: "Formatted at 2026-09-29 20:53:26 on dist-test-slave-m9g1"
server_key: "c5d7f50bf3a5082caf22c7482ca82c4a"
server_key_iv: "8846eebb8b77da9055e2b3f07d44ddff"
server_key_version: "encryptionkey@0"
I20260929 20:53:26.658887 4993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:53:26.662025 5007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:26.663415 4993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:53:26.663571 4993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "1844967032464e31bf72323ba5c4d7b0"
format_stamp: "Formatted at 2026-09-29 20:53:26 on dist-test-slave-m9g1"
server_key: "c5d7f50bf3a5082caf22c7482ca82c4a"
server_key_iv: "8846eebb8b77da9055e2b3f07d44ddff"
server_key_version: "encryptionkey@0"
I20260929 20:53:26.663784 4993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:26.669258 4993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:26.670001 4993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:26.670296 4993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:26.686612 4993 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38957
I20260929 20:53:26.686642 5058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38957 every 8 connection(s)
I20260929 20:53:26.687889 4993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:53:26.691519 5059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:26.692233 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 4993
I20260929 20:53:26.692420 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:53:26.693172 388 external_mini_cluster.cc:1428] Setting key effddf21d98f22068508ed6206820660
I20260929 20:53:26.701325 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:26.695346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57628 (local address 127.0.97.62:38957)
0929 20:53:26.696357 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:26.696368 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:26.696409 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:53:26.697919 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:26.697941 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:26.698419 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:26.698786 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:26.699761 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:26.699767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:26.699780 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:53:26.699793 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:26.699860 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:53:26.699876 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:26.700069 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:26.700190 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 20:53:26.700427 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":356,"threads_started":1}
I20260929 20:53:26.704160 5059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0: Bootstrap starting.
I20260929 20:53:26.706975 5059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:26.707917 5059 log.cc:824] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:26.710498 5059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0: No bootstrap required, opened a new log
I20260929 20:53:26.717213 5059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1844967032464e31bf72323ba5c4d7b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38957 } }
I20260929 20:53:26.717593 5059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:26.717682 5059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1844967032464e31bf72323ba5c4d7b0, State: Initialized, Role: FOLLOWER
I20260929 20:53:26.718250 5059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1844967032464e31bf72323ba5c4d7b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38957 } }
I20260929 20:53:26.718464 5059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:53:26.718580 5059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:53:26.718724 5059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:26.720731 5059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1844967032464e31bf72323ba5c4d7b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38957 } }
I20260929 20:53:26.721179 5059 leader_election.cc:302] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1844967032464e31bf72323ba5c4d7b0; no voters:
I20260929 20:53:26.721724 5059 leader_election.cc:288] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:53:26.721840 5064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:26.722548 5064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0 [term 1 LEADER]: Becoming Leader. State: Replica: 1844967032464e31bf72323ba5c4d7b0, State: Running, Role: LEADER
I20260929 20:53:26.723038 5064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1844967032464e31bf72323ba5c4d7b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38957 } }
I20260929 20:53:26.723660 5059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:53:26.725461 5066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1844967032464e31bf72323ba5c4d7b0. Latest consensus state: current_term: 1 leader_uuid: "1844967032464e31bf72323ba5c4d7b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1844967032464e31bf72323ba5c4d7b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38957 } } }
I20260929 20:53:26.725797 5066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:26.726457 5065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1844967032464e31bf72323ba5c4d7b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1844967032464e31bf72323ba5c4d7b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38957 } } }
I20260929 20:53:26.726842 5065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:26.736577 5070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:53:26.741767 5070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:53:26.750504 5070 catalog_manager.cc:1347] Generated new cluster ID: 16e446be88bd48f5a5be6b0977e45335
I20260929 20:53:26.750623 5070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:53:26.776342 5070 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:53:26.776826 5070 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:53:26.791076 5070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0: Generated new TSK 0
I20260929 20:53:26.792143 5070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:53:26.796921 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38957
--builtin_ntp_servers=127.0.97.20:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 20:53:27.104334 5083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:27.104601 5083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:27.104729 5083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:27.111709 5083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:27.111820 5083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:27.111981 5083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:53:27.120630 5083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:27.121829 5083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:27.123633 5083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:27.135813 5088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:27.135748 5091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:27.135787 5089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:27.136512 5083 server_base.cc:1034] running on GCE node
I20260929 20:53:27.137094 5083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:27.137971 5083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:27.139173 5083 hybrid_clock.cc:648] HybridClock initialized: now 1790715207139088 us; error 71 us; skew 500 ppm
I20260929 20:53:27.139465 5083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:53:27.141477 5083 webserver.cc:460] Webserver started at http://127.0.97.1:46509/ using document root <none> and password file <none>
I20260929 20:53:27.142105 5083 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:27.142220 5083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:27.142537 5083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:27.145202 5083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "8d6986352d154b31b6f0457fea5db822"
format_stamp: "Formatted at 2026-09-29 20:53:27 on dist-test-slave-m9g1"
server_key: "9b1ce0044de0e665944c308c08dde210"
server_key_iv: "c6e66edcff20b9a76978bc1d34b2a5c8"
server_key_version: "encryptionkey@0"
I20260929 20:53:27.145941 5083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8d6986352d154b31b6f0457fea5db822"
format_stamp: "Formatted at 2026-09-29 20:53:27 on dist-test-slave-m9g1"
server_key: "9b1ce0044de0e665944c308c08dde210"
server_key_iv: "c6e66edcff20b9a76978bc1d34b2a5c8"
server_key_version: "encryptionkey@0"
I20260929 20:53:27.150815 5083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:53:27.154099 5097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:27.155386 5083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:53:27.155536 5083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "8d6986352d154b31b6f0457fea5db822"
format_stamp: "Formatted at 2026-09-29 20:53:27 on dist-test-slave-m9g1"
server_key: "9b1ce0044de0e665944c308c08dde210"
server_key_iv: "c6e66edcff20b9a76978bc1d34b2a5c8"
server_key_version: "encryptionkey@0"
I20260929 20:53:27.155745 5083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:27.160976 5083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:27.161724 5083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:27.161950 5083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:27.162851 5083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:27.164122 5083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:27.164213 5083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:27.164314 5083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:27.164379 5083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:27.193265 5083 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38523
I20260929 20:53:27.193277 5209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38523 every 8 connection(s)
I20260929 20:53:27.194695 5083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:53:27.203084 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:27.197344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59903 (local address 127.0.97.62:38957)
0929 20:53:27.197520 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:27.197525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:27.199063 (+ 1538us) server_negotiation.cc:407] Connection header received
0929 20:53:27.200877 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:27.200883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:27.200950 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:27.201049 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:27.202403 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:27.202410 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:27.202414 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:53:27.202432 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:27.202464 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:53:27.202476 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:27.202576 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:27.202749 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 20:53:27.202816 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:53:27.203701 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 5083
I20260929 20:53:27.203866 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:53:27.204165 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:27.197942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38957 (local address 127.0.97.1:59903)
0929 20:53:27.198896 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:27.198934 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:27.200594 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:27.201301 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:27.201311 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:27.201411 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:27.202052 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:53:27.202055 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:27.202100 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:53:27.202239 (+ 139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:27.202555 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:27.202564 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:27.202571 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:53:27.202825 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 20:53:27.203094 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":283,"threads_started":1}
I20260929 20:53:27.204505 388 external_mini_cluster.cc:1428] Setting key b136ca2e67cacc4fbe661aa622f7c83a
I20260929 20:53:27.205694 5210 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38957
I20260929 20:53:27.206014 5210 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:27.206771 5210 heartbeater.cc:511] Master 127.0.97.62:38957 requested a full tablet report, sending...
I20260929 20:53:27.207242 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38957
--builtin_ntp_servers=127.0.97.20:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 20:53:27.208649 5024 ts_manager.cc:194] Registered new tserver with Master: 8d6986352d154b31b6f0457fea5db822 (127.0.97.1:38523)
W20260929 20:53:27.517957 5213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:27.518260 5213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:27.518395 5213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:27.525643 5213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:27.525732 5213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:27.525872 5213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:53:27.534633 5213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:27.535794 5213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:27.537530 5213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:27.549706 5218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:27.549669 5219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:27.549634 5221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:27.550608 5213 server_base.cc:1034] running on GCE node
I20260929 20:53:27.551213 5213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:27.552101 5213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:27.553311 5213 hybrid_clock.cc:648] HybridClock initialized: now 1790715207553267 us; error 37 us; skew 500 ppm
I20260929 20:53:27.553607 5213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:53:27.555670 5213 webserver.cc:460] Webserver started at http://127.0.97.2:36203/ using document root <none> and password file <none>
I20260929 20:53:27.556300 5213 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:27.556393 5213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:27.556679 5213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:27.559383 5213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "0ffb2711a2da47ffbb7e1589f13ec4b0"
format_stamp: "Formatted at 2026-09-29 20:53:27 on dist-test-slave-m9g1"
server_key: "84755e8792a7bfac6038f10ce3c8dcb9"
server_key_iv: "09ff0721bfda5a4cf108aaed58c233eb"
server_key_version: "encryptionkey@0"
I20260929 20:53:27.560127 5213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0ffb2711a2da47ffbb7e1589f13ec4b0"
format_stamp: "Formatted at 2026-09-29 20:53:27 on dist-test-slave-m9g1"
server_key: "84755e8792a7bfac6038f10ce3c8dcb9"
server_key_iv: "09ff0721bfda5a4cf108aaed58c233eb"
server_key_version: "encryptionkey@0"
I20260929 20:53:27.564996 5213 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:53:27.568281 5227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:27.569473 5213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:53:27.569622 5213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "0ffb2711a2da47ffbb7e1589f13ec4b0"
format_stamp: "Formatted at 2026-09-29 20:53:27 on dist-test-slave-m9g1"
server_key: "84755e8792a7bfac6038f10ce3c8dcb9"
server_key_iv: "09ff0721bfda5a4cf108aaed58c233eb"
server_key_version: "encryptionkey@0"
I20260929 20:53:27.569809 5213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:27.575054 5213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:27.575748 5213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:27.575968 5213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:27.576767 5213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:27.577979 5213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:27.578069 5213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:27.578166 5213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:27.578254 5213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 20:53:27.606830 5213 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40191
I20260929 20:53:27.606876 5339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40191 every 8 connection(s)
I20260929 20:53:27.608078 5213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:53:27.614624 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 5213
I20260929 20:53:27.614778 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:53:27.615259 388 external_mini_cluster.cc:1428] Setting key ae5f74adb88d95864a12db26c9e2f693
I20260929 20:53:27.615777 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:27.610444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40243 (local address 127.0.97.62:38957)
0929 20:53:27.610605 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:27.610609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:27.611934 (+ 1325us) server_negotiation.cc:407] Connection header received
0929 20:53:27.613517 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:27.613522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:27.613577 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:27.613658 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:27.615078 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:27.615084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:27.615089 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:53:27.615109 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:27.615147 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:53:27.615163 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:27.615285 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:27.615433 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 20:53:27.615509 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:53:27.616916 5342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:27.610949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38957 (local address 127.0.97.2:40243)
0929 20:53:27.611813 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:27.611849 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:27.613292 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:27.613841 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:27.613848 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:27.613941 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:27.614730 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:53:27.614733 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:27.614791 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:53:27.614900 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:27.615251 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:27.615257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:27.615262 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:53:27.615502 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 20:53:27.615753 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":257,"threads_started":1}
I20260929 20:53:27.617986 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38957
--builtin_ntp_servers=127.0.97.20:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 20:53:27.618280 5340 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38957
I20260929 20:53:27.618574 5340 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:27.619240 5340 heartbeater.cc:511] Master 127.0.97.62:38957 requested a full tablet report, sending...
I20260929 20:53:27.620643 5024 ts_manager.cc:194] Registered new tserver with Master: 0ffb2711a2da47ffbb7e1589f13ec4b0 (127.0.97.2:40191)
W20260929 20:53:27.924823 5343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:27.925099 5343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:27.925210 5343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:27.932250 5343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:27.932353 5343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:27.932482 5343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:53:27.941470 5343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:27.942750 5343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:27.944422 5343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:27.956548 5348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:27.956606 5349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:27.956585 5351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:27.957175 5343 server_base.cc:1034] running on GCE node
I20260929 20:53:27.957742 5343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:27.958666 5343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:27.959851 5343 hybrid_clock.cc:648] HybridClock initialized: now 1790715207959804 us; error 46 us; skew 500 ppm
I20260929 20:53:27.960146 5343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:53:27.962131 5343 webserver.cc:460] Webserver started at http://127.0.97.3:46785/ using document root <none> and password file <none>
I20260929 20:53:27.962771 5343 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:27.962864 5343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:27.963148 5343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:27.965761 5343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "cb9353ca71664be78653d2ad98488474"
format_stamp: "Formatted at 2026-09-29 20:53:27 on dist-test-slave-m9g1"
server_key: "996ea6c882c578209228c16d154e6b77"
server_key_iv: "337a0babea1ae878ec37d245cb6aa0a8"
server_key_version: "encryptionkey@0"
I20260929 20:53:27.966517 5343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cb9353ca71664be78653d2ad98488474"
format_stamp: "Formatted at 2026-09-29 20:53:27 on dist-test-slave-m9g1"
server_key: "996ea6c882c578209228c16d154e6b77"
server_key_iv: "337a0babea1ae878ec37d245cb6aa0a8"
server_key_version: "encryptionkey@0"
I20260929 20:53:27.971283 5343 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:53:27.974505 5357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:27.975706 5343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:53:27.975848 5343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "cb9353ca71664be78653d2ad98488474"
format_stamp: "Formatted at 2026-09-29 20:53:27 on dist-test-slave-m9g1"
server_key: "996ea6c882c578209228c16d154e6b77"
server_key_iv: "337a0babea1ae878ec37d245cb6aa0a8"
server_key_version: "encryptionkey@0"
I20260929 20:53:27.976047 5343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:27.981143 5343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:27.981842 5343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:27.982075 5343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:27.982933 5343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:27.984177 5343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:27.984266 5343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:27.984368 5343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:27.984436 5343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:28.012552 5343 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34653
I20260929 20:53:28.012591 5469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34653 every 8 connection(s)
I20260929 20:53:28.013803 5343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:53:28.014391 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 5343
I20260929 20:53:28.014559 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:53:28.015115 388 external_mini_cluster.cc:1428] Setting key b3448ce2a8ef520ab802eb473f64415d
I20260929 20:53:28.021498 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.016464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59451 (local address 127.0.97.62:38957)
0929 20:53:28.016658 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:28.016663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:28.018238 (+ 1575us) server_negotiation.cc:407] Connection header received
0929 20:53:28.019778 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:28.019782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:28.019829 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:28.019907 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:28.021007 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:28.021011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:28.021014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:53:28.021028 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:28.021056 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:53:28.021067 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:28.021155 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:28.021271 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 20:53:28.021314 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:53:28.022440 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.017110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38957 (local address 127.0.97.3:59451)
0929 20:53:28.018100 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:28.018135 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:28.019574 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:28.020086 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:28.020094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:28.020184 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:28.020757 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:53:28.020760 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:28.020797 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:53:28.020875 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:28.021133 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:28.021138 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:28.021143 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:53:28.021342 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 20:53:28.021524 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":290,"threads_started":1}
I20260929 20:53:28.023619 5470 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38957
I20260929 20:53:28.023871 5470 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:28.024432 5470 heartbeater.cc:511] Master 127.0.97.62:38957 requested a full tablet report, sending...
I20260929 20:53:28.025619 5024 ts_manager.cc:194] Registered new tserver with Master: cb9353ca71664be78653d2ad98488474 (127.0.97.3:34653)
I20260929 20:53:28.029837 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:53:28.035900 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.033811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.0.97.62:38957)
0929 20:53:28.033943 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:28.033947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:28.034406 (+ 459us) server_negotiation.cc:407] Connection header received
0929 20:53:28.034526 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:28.034529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:28.034572 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:28.034645 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:28.035464 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:28.035467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:28.035470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:53:28.035483 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:28.035512 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:53:28.035522 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:28.035606 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:28.035656 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 20:53:28.035696 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 20:53:28.039417 5024 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:53:28.045878 5024 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:53:28.064041 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.057400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34653 (local address 127.0.0.1:39580)
0929 20:53:28.060086 (+ 2686us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:28.060102 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:28.060219 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:28.061152 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:28.061160 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:28.061251 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:28.062662 (+ 1411us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:53:28.062665 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:28.062694 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:53:28.062778 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:28.063269 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:28.063278 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:28.063285 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:53:28.063461 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 20:53:28.063562 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2559,"thread_start_us":2344,"threads_started":1}
I20260929 20:53:28.064714 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.058048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39580 (local address 127.0.97.3:34653)
0929 20:53:28.060341 (+ 2293us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:28.060369 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:28.060400 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:53:28.060467 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:28.060474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:28.060727 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:28.060856 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:28.062949 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:28.062957 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:28.062972 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:53:28.062990 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:28.063047 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:53:28.063070 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:28.063219 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:28.064260 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0929 20:53:28.064369 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2140,"thread_start_us":1987,"threads_started":1}
I20260929 20:53:28.065135 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.057228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40806 (local address 127.0.97.2:40191)
0929 20:53:28.058099 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:28.058109 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:28.059722 (+ 1613us) server_negotiation.cc:407] Connection header received
0929 20:53:28.060477 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:28.060484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:28.060728 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:28.060849 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:28.064233 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:28.064241 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:28.064253 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:53:28.064271 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:28.064327 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:53:28.064350 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:28.064480 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:28.064710 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 20:53:28.064811 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":497,"threads_started":1}
I20260929 20:53:28.066366 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.057064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40191 (local address 127.0.0.1:40806)
0929 20:53:28.059581 (+ 2517us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:28.059616 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:28.060347 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:28.063286 (+ 2939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:28.063290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:28.063311 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:28.064011 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:53:28.064014 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:28.064033 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:53:28.064073 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:28.064567 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:28.064572 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:28.064577 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:53:28.065382 (+ 805us) client_negotiation.cc:240] Negotiation successful
0929 20:53:28.065432 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2288,"thread_start_us":2142,"threads_started":1}
I20260929 20:53:28.068531 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.060157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38523 (local address 127.0.0.1:40416)
0929 20:53:28.065067 (+ 4910us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:28.065083 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:28.065153 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:28.066817 (+ 1664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:28.066821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:28.066864 (+ 43us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:28.067312 (+ 448us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:53:28.067315 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:28.067330 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:53:28.067381 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:28.068149 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:28.068154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:28.068159 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:53:28.068265 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 20:53:28.068312 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4819,"thread_start_us":6546,"threads_started":1}
I20260929 20:53:28.068661 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.060596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.0.97.1:38523)
0929 20:53:28.065917 (+ 5321us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:28.065925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:28.065952 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:53:28.066013 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:28.066022 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:28.066291 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:28.067053 (+ 762us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:28.067876 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:28.067883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:28.067895 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:53:28.067917 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:28.067968 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:53:28.067987 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:28.068183 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:28.068283 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 20:53:28.068369 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5170,"thread_start_us":5069,"threads_started":1}
I20260929 20:53:28.070470 5405 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3edf8e85fb415683e30213fc527411 (DEFAULT_TABLE table=test-table [id=abd7b3f1d5f645f7a68f3b0ad736ecce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:28.070436 5275 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3edf8e85fb415683e30213fc527411 (DEFAULT_TABLE table=test-table [id=abd7b3f1d5f645f7a68f3b0ad736ecce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:28.071681 5405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3edf8e85fb415683e30213fc527411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:28.071743 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3edf8e85fb415683e30213fc527411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:28.071859 5145 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3edf8e85fb415683e30213fc527411 (DEFAULT_TABLE table=test-table [id=abd7b3f1d5f645f7a68f3b0ad736ecce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:28.073071 5145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3edf8e85fb415683e30213fc527411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:28.083391 5488 tablet_bootstrap.cc:492] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474: Bootstrap starting.
I20260929 20:53:28.086378 5488 tablet_bootstrap.cc:654] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:28.087283 5488 log.cc:824] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:28.091240 5489 tablet_bootstrap.cc:492] T 7d3edf8e85fb415683e30213fc527411 P 8d6986352d154b31b6f0457fea5db822: Bootstrap starting.
I20260929 20:53:28.092041 5490 tablet_bootstrap.cc:492] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0: Bootstrap starting.
I20260929 20:53:28.094681 5489 tablet_bootstrap.cc:654] T 7d3edf8e85fb415683e30213fc527411 P 8d6986352d154b31b6f0457fea5db822: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:28.095256 5490 tablet_bootstrap.cc:654] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:28.095691 5489 log.cc:824] T 7d3edf8e85fb415683e30213fc527411 P 8d6986352d154b31b6f0457fea5db822: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:28.096349 5490 log.cc:824] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:28.096804 5488 tablet_bootstrap.cc:492] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474: No bootstrap required, opened a new log
I20260929 20:53:28.097098 5488 ts_tablet_manager.cc:1397] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20260929 20:53:28.098090 5489 tablet_bootstrap.cc:492] T 7d3edf8e85fb415683e30213fc527411 P 8d6986352d154b31b6f0457fea5db822: No bootstrap required, opened a new log
I20260929 20:53:28.098388 5489 ts_tablet_manager.cc:1397] T 7d3edf8e85fb415683e30213fc527411 P 8d6986352d154b31b6f0457fea5db822: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 20:53:28.099257 5490 tablet_bootstrap.cc:492] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0: No bootstrap required, opened a new log
I20260929 20:53:28.099495 5490 ts_tablet_manager.cc:1397] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 20:53:28.104962 5489 raft_consensus.cc:348] T 7d3edf8e85fb415683e30213fc527411 P 8d6986352d154b31b6f0457fea5db822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffb2711a2da47ffbb7e1589f13ec4b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40191 } } peers { permanent_uuid: "cb9353ca71664be78653d2ad98488474" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34653 } } peers { permanent_uuid: "8d6986352d154b31b6f0457fea5db822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38523 } }
I20260929 20:53:28.105283 5489 raft_consensus.cc:374] T 7d3edf8e85fb415683e30213fc527411 P 8d6986352d154b31b6f0457fea5db822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:28.105365 5489 raft_consensus.cc:729] T 7d3edf8e85fb415683e30213fc527411 P 8d6986352d154b31b6f0457fea5db822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6986352d154b31b6f0457fea5db822, State: Initialized, Role: FOLLOWER
I20260929 20:53:28.105903 5489 consensus_queue.cc:260] T 7d3edf8e85fb415683e30213fc527411 P 8d6986352d154b31b6f0457fea5db822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffb2711a2da47ffbb7e1589f13ec4b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40191 } } peers { permanent_uuid: "cb9353ca71664be78653d2ad98488474" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34653 } } peers { permanent_uuid: "8d6986352d154b31b6f0457fea5db822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38523 } }
I20260929 20:53:28.106721 5488 raft_consensus.cc:348] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffb2711a2da47ffbb7e1589f13ec4b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40191 } } peers { permanent_uuid: "cb9353ca71664be78653d2ad98488474" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34653 } } peers { permanent_uuid: "8d6986352d154b31b6f0457fea5db822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38523 } }
I20260929 20:53:28.107146 5488 raft_consensus.cc:374] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:28.107263 5488 raft_consensus.cc:729] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9353ca71664be78653d2ad98488474, State: Initialized, Role: FOLLOWER
I20260929 20:53:28.107923 5488 consensus_queue.cc:260] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffb2711a2da47ffbb7e1589f13ec4b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40191 } } peers { permanent_uuid: "cb9353ca71664be78653d2ad98488474" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34653 } } peers { permanent_uuid: "8d6986352d154b31b6f0457fea5db822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38523 } }
I20260929 20:53:28.108422 5210 heartbeater.cc:503] Master 127.0.97.62:38957 was elected leader, sending a full tablet report...
I20260929 20:53:28.109028 5489 ts_tablet_manager.cc:1428] T 7d3edf8e85fb415683e30213fc527411 P 8d6986352d154b31b6f0457fea5db822: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260929 20:53:28.108901 5490 raft_consensus.cc:348] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffb2711a2da47ffbb7e1589f13ec4b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40191 } } peers { permanent_uuid: "cb9353ca71664be78653d2ad98488474" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34653 } } peers { permanent_uuid: "8d6986352d154b31b6f0457fea5db822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38523 } }
I20260929 20:53:28.109350 5490 raft_consensus.cc:374] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:28.109479 5490 raft_consensus.cc:729] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ffb2711a2da47ffbb7e1589f13ec4b0, State: Initialized, Role: FOLLOWER
I20260929 20:53:28.110114 5490 consensus_queue.cc:260] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffb2711a2da47ffbb7e1589f13ec4b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40191 } } peers { permanent_uuid: "cb9353ca71664be78653d2ad98488474" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34653 } } peers { permanent_uuid: "8d6986352d154b31b6f0457fea5db822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38523 } }
I20260929 20:53:28.114215 5340 heartbeater.cc:503] Master 127.0.97.62:38957 was elected leader, sending a full tablet report...
I20260929 20:53:28.115089 5490 ts_tablet_manager.cc:1428] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260929 20:53:28.115883 5470 heartbeater.cc:503] Master 127.0.97.62:38957 was elected leader, sending a full tablet report...
I20260929 20:53:28.116551 5488 ts_tablet_manager.cc:1428] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474: Time spent starting tablet: real 0.019s user 0.014s sys 0.003s
I20260929 20:53:28.118224 5496 raft_consensus.cc:482] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:53:28.118427 5496 raft_consensus.cc:504] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffb2711a2da47ffbb7e1589f13ec4b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40191 } } peers { permanent_uuid: "cb9353ca71664be78653d2ad98488474" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34653 } } peers { permanent_uuid: "8d6986352d154b31b6f0457fea5db822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38523 } }
I20260929 20:53:28.119768 5496 leader_election.cc:288] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb9353ca71664be78653d2ad98488474 (127.0.97.3:34653), 8d6986352d154b31b6f0457fea5db822 (127.0.97.1:38523)
I20260929 20:53:28.122499 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.120011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39553 (local address 127.0.97.3:34653)
0929 20:53:28.120155 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:28.120159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:28.120579 (+ 420us) server_negotiation.cc:407] Connection header received
0929 20:53:28.120734 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:28.120740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:28.120798 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:28.120901 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:28.121953 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:28.121957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:28.121961 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:53:28.121975 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:28.122002 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:53:28.122012 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:28.122109 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:28.122206 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 20:53:28.122258 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 20:53:28.122640 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.120070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34653 (local address 127.0.97.2:39553)
0929 20:53:28.120489 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:28.120504 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:28.120600 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:28.121054 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:28.121057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:28.121076 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:28.121763 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:53:28.121767 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:28.121787 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:53:28.121828 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:28.122089 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:28.122094 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:28.122099 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:53:28.122213 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 20:53:28.122258 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":232,"threads_started":1}
I20260929 20:53:28.123216 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.120842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54191 (local address 127.0.97.1:38523)
0929 20:53:28.121004 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:28.121009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:28.121213 (+ 204us) server_negotiation.cc:407] Connection header received
0929 20:53:28.121355 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:28.121359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:28.121409 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:28.121488 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:28.122549 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:28.122554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:28.122558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:53:28.122576 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:28.122608 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:53:28.122622 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:28.122700 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:28.122911 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 20:53:28.122965 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:53:28.123338 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.120627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38523 (local address 127.0.97.2:54191)
0929 20:53:28.121125 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:28.121140 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:28.121237 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:28.121624 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:28.121629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:28.121650 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:28.122323 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:53:28.122336 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:28.122351 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:53:28.122395 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:28.122781 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:28.122785 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:28.122789 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 20:53:28.122940 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 20:53:28.122983 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":230,"threads_started":1}
I20260929 20:53:28.123574 5425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3edf8e85fb415683e30213fc527411" candidate_uuid: "0ffb2711a2da47ffbb7e1589f13ec4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9353ca71664be78653d2ad98488474" is_pre_election: true
I20260929 20:53:28.124063 5425 raft_consensus.cc:2454] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ffb2711a2da47ffbb7e1589f13ec4b0 in term 0.
I20260929 20:53:28.124295 5165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3edf8e85fb415683e30213fc527411" candidate_uuid: "0ffb2711a2da47ffbb7e1589f13ec4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6986352d154b31b6f0457fea5db822" is_pre_election: true
I20260929 20:53:28.124715 5229 leader_election.cc:302] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ffb2711a2da47ffbb7e1589f13ec4b0, cb9353ca71664be78653d2ad98488474; no voters:
I20260929 20:53:28.124754 5165 raft_consensus.cc:2454] T 7d3edf8e85fb415683e30213fc527411 P 8d6986352d154b31b6f0457fea5db822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ffb2711a2da47ffbb7e1589f13ec4b0 in term 0.
I20260929 20:53:28.125036 5496 raft_consensus.cc:2781] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:53:28.125200 5496 raft_consensus.cc:482] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:53:28.125294 5496 raft_consensus.cc:3037] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:28.127437 5496 raft_consensus.cc:504] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffb2711a2da47ffbb7e1589f13ec4b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40191 } } peers { permanent_uuid: "cb9353ca71664be78653d2ad98488474" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34653 } } peers { permanent_uuid: "8d6986352d154b31b6f0457fea5db822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38523 } }
I20260929 20:53:28.127942 5496 leader_election.cc:288] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [CANDIDATE]: Term 1 election: Requested vote from peers cb9353ca71664be78653d2ad98488474 (127.0.97.3:34653), 8d6986352d154b31b6f0457fea5db822 (127.0.97.1:38523)
I20260929 20:53:28.128434 5425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3edf8e85fb415683e30213fc527411" candidate_uuid: "0ffb2711a2da47ffbb7e1589f13ec4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9353ca71664be78653d2ad98488474"
I20260929 20:53:28.128468 5165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3edf8e85fb415683e30213fc527411" candidate_uuid: "0ffb2711a2da47ffbb7e1589f13ec4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6986352d154b31b6f0457fea5db822"
I20260929 20:53:28.128599 5425 raft_consensus.cc:3037] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:28.128633 5165 raft_consensus.cc:3037] T 7d3edf8e85fb415683e30213fc527411 P 8d6986352d154b31b6f0457fea5db822 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:28.130790 5425 raft_consensus.cc:2454] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ffb2711a2da47ffbb7e1589f13ec4b0 in term 1.
I20260929 20:53:28.130877 5165 raft_consensus.cc:2454] T 7d3edf8e85fb415683e30213fc527411 P 8d6986352d154b31b6f0457fea5db822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ffb2711a2da47ffbb7e1589f13ec4b0 in term 1.
I20260929 20:53:28.131160 5229 leader_election.cc:302] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ffb2711a2da47ffbb7e1589f13ec4b0, cb9353ca71664be78653d2ad98488474; no voters:
I20260929 20:53:28.131404 5496 raft_consensus.cc:2781] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:28.131796 5496 raft_consensus.cc:686] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [term 1 LEADER]: Becoming Leader. State: Replica: 0ffb2711a2da47ffbb7e1589f13ec4b0, State: Running, Role: LEADER
I20260929 20:53:28.132283 5496 consensus_queue.cc:237] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffb2711a2da47ffbb7e1589f13ec4b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40191 } } peers { permanent_uuid: "cb9353ca71664be78653d2ad98488474" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34653 } } peers { permanent_uuid: "8d6986352d154b31b6f0457fea5db822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38523 } }
I20260929 20:53:28.136791 5024 catalog_manager.cc:5495] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ffb2711a2da47ffbb7e1589f13ec4b0 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0ffb2711a2da47ffbb7e1589f13ec4b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffb2711a2da47ffbb7e1589f13ec4b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb9353ca71664be78653d2ad98488474" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6986352d154b31b6f0457fea5db822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38523 } health_report { overall_health: UNKNOWN } } }
I20260929 20:53:28.165588 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.163964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40812 (local address 127.0.97.2:40191)
0929 20:53:28.164126 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:28.164131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:28.164162 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:53:28.164252 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:28.164256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:28.164329 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:28.164398 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:28.165134 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:28.165137 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:28.165140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:53:28.165152 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:28.165176 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:53:28.165186 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:28.165284 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:28.165319 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 20:53:28.165359 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:53:28.173148 5165 raft_consensus.cc:1261] T 7d3edf8e85fb415683e30213fc527411 P 8d6986352d154b31b6f0457fea5db822 [term 1 FOLLOWER]: Refusing update from remote peer 0ffb2711a2da47ffbb7e1589f13ec4b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:28.173198 5425 raft_consensus.cc:1261] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474 [term 1 FOLLOWER]: Refusing update from remote peer 0ffb2711a2da47ffbb7e1589f13ec4b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:28.174027 5499 consensus_queue.cc:1035] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6986352d154b31b6f0457fea5db822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:28.174372 5496 consensus_queue.cc:1035] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9353ca71664be78653d2ad98488474" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:28.197535 5024 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:28.197800 5024 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:28.201452 5024 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1844967032464e31bf72323ba5c4d7b0: Sending DeleteTablet for 3 replicas of tablet 7d3edf8e85fb415683e30213fc527411
I20260929 20:53:28.202527 5275 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3edf8e85fb415683e30213fc527411 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:28 UTC) from {username='slave'} at 127.0.0.1:40806
I20260929 20:53:28.202589 5405 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3edf8e85fb415683e30213fc527411 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:28 UTC) from {username='slave'} at 127.0.0.1:39580
I20260929 20:53:28.204123 5145 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3edf8e85fb415683e30213fc527411 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:28 UTC) from {username='slave'} at 127.0.0.1:40416
I20260929 20:53:28.212709 5514 tablet_replica.cc:330] T 7d3edf8e85fb415683e30213fc527411 P 8d6986352d154b31b6f0457fea5db822: stopping tablet replica
I20260929 20:53:28.212738 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 5083
I20260929 20:53:28.215981 5514 raft_consensus.cc:2229] T 7d3edf8e85fb415683e30213fc527411 P 8d6986352d154b31b6f0457fea5db822 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:28.216763 5512 tablet_replica.cc:330] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474: stopping tablet replica
I20260929 20:53:28.217199 5512 raft_consensus.cc:2229] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:28.217689 5514 raft_consensus.cc:2258] T 7d3edf8e85fb415683e30213fc527411 P 8d6986352d154b31b6f0457fea5db822 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:28.217685 5512 raft_consensus.cc:2258] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:28.218343 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.213736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40424 (local address 127.0.97.1:38523)
0929 20:53:28.213907 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:28.213913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:28.216144 (+ 2231us) server_negotiation.cc:407] Connection header received
0929 20:53:28.216348 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:28.216354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:28.216450 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:28.216538 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:28.217603 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:28.217608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:28.217613 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:53:28.217631 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:28.217667 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:53:28.217686 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:28.217776 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:28.217993 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 20:53:28.218051 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:53:28.219328 5205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:28.219899 5512 ts_tablet_manager.cc:1905] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:28.221056 5513 tablet_replica.cc:330] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0: stopping tablet replica
I20260929 20:53:28.221575 5513 raft_consensus.cc:2229] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:53:28.222286 5513 raft_consensus.cc:2258] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:28.225133 5513 ts_tablet_manager.cc:1905] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:28.227247 5512 ts_tablet_manager.cc:1918] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:28.227458 5512 log.cc:1196] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/7d3edf8e85fb415683e30213fc527411
I20260929 20:53:28.228010 5512 ts_tablet_manager.cc:1939] T 7d3edf8e85fb415683e30213fc527411 P cb9353ca71664be78653d2ad98488474: Deleting consensus metadata
I20260929 20:53:28.229875 5009 catalog_manager.cc:4841] TS cb9353ca71664be78653d2ad98488474 (127.0.97.3:34653): tablet 7d3edf8e85fb415683e30213fc527411 (table test-table [id=abd7b3f1d5f645f7a68f3b0ad736ecce]) successfully deleted
I20260929 20:53:28.233472 5513 ts_tablet_manager.cc:1918] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:28.233639 5513 log.cc:1196] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/7d3edf8e85fb415683e30213fc527411
I20260929 20:53:28.234127 5513 ts_tablet_manager.cc:1939] T 7d3edf8e85fb415683e30213fc527411 P 0ffb2711a2da47ffbb7e1589f13ec4b0: Deleting consensus metadata
I20260929 20:53:28.235893 5010 catalog_manager.cc:4841] TS 0ffb2711a2da47ffbb7e1589f13ec4b0 (127.0.97.2:40191): tablet 7d3edf8e85fb415683e30213fc527411 (table test-table [id=abd7b3f1d5f645f7a68f3b0ad736ecce]) successfully deleted
I20260929 20:53:28.314527 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 5083
I20260929 20:53:28.328285 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 5213
W20260929 20:53:28.328390 5009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38523 (error 108)
I20260929 20:53:28.330467 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.330031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38523 (local address 127.0.0.1:40434)
0929 20:53:28.330218 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:28.330328 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 20:53:28.330687 5009 catalog_manager.cc:4584] TS 8d6986352d154b31b6f0457fea5db822 (127.0.97.1:38523): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d3edf8e85fb415683e30213fc527411: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38523: connect: Connection refused (error 111)
I20260929 20:53:28.331528 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.329073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40818 (local address 127.0.97.2:40191)
0929 20:53:28.329255 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:28.329262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:28.329283 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:53:28.329363 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:28.329369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:28.329429 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:28.329536 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:28.330913 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:28.330919 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:28.330924 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:53:28.330945 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:28.330974 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:53:28.330985 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:28.331111 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:28.331230 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 20:53:28.331274 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:53:28.332230 5335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:28.384297 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.383770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38523 (local address 127.0.0.1:40436)
0929 20:53:28.384024 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:28.384129 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 20:53:28.385257 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.384889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38523 (local address 127.0.0.1:40444)
0929 20:53:28.385023 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:28.385112 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 20:53:28.437919 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 5213
I20260929 20:53:28.445569 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.445138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38523 (local address 127.0.0.1:40458)
0929 20:53:28.445344 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:28.445434 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 20:53:28.446648 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.446309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38523 (local address 127.0.0.1:40468)
0929 20:53:28.446430 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:28.446502 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 20:53:28.453020 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 5343
I20260929 20:53:28.455562 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.453780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39592 (local address 127.0.97.3:34653)
0929 20:53:28.453959 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:28.453964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:28.453981 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:53:28.454130 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:28.454134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:28.454252 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:28.454334 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:28.455039 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:28.455042 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:28.455046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:53:28.455060 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:28.455103 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:53:28.455115 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:28.455216 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:28.455249 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 20:53:28.455291 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:53:28.456146 5465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:28.527096 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.526648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38523 (local address 127.0.0.1:40482)
0929 20:53:28.526870 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:28.526956 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 20:53:28.528285 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:28.527782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38523 (local address 127.0.0.1:40490)
0929 20:53:28.528052 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:28.528128 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260929 20:53:28.541949 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 5343
I20260929 20:53:28.554356 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 4993
I20260929 20:53:28.555392 5054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:28.681005 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 4993
2026-09-29T20:53:28Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2442 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 20:53:28.717002 388 mini_oidc.cc:152] Starting JWKS server
I20260929 20:53:28.719497 388 webserver.cc:460] Webserver started at https://127.0.0.1:34275/ using document root <none> and password file <none>
I20260929 20:53:28.719802 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 20:53:28.720362 388 webserver.cc:460] Webserver started at http://127.0.0.1:34565/ using document root <none> and password file <none>
2026-09-29T20:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:53:28Z Disabled control of system clock
I20260929 20:53:28.743283 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40511
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40511
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34275/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0/testchainca.cert with env {}
W20260929 20:53:29.049314 5526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:29.049606 5526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:29.049691 5526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:29.056875 5526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:53:29.056985 5526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:29.057021 5526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:29.057077 5526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:53:29.057119 5526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:53:29.057178 5526 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:53:29.057214 5526 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34275/jwks.json
I20260929 20:53:29.067307 5526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40511
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34275/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:29.068584 5526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:29.069794 5526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:29.081040 5532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:29.081015 5531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:29.081106 5534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:29.081797 5526 server_base.cc:1034] running on GCE node
I20260929 20:53:29.082643 5526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:29.084028 5526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:29.085218 5526 hybrid_clock.cc:648] HybridClock initialized: now 1790715209085168 us; error 45 us; skew 500 ppm
I20260929 20:53:29.085542 5526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:53:29.087713 5526 webserver.cc:460] Webserver started at http://127.0.97.62:36341/ using document root <none> and password file <none>
I20260929 20:53:29.088399 5526 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:29.088506 5526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:29.088791 5526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:29.091632 5526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "3882a14ace534ec2a9cd849da1d8d2af"
format_stamp: "Formatted at 2026-09-29 20:53:29 on dist-test-slave-m9g1"
server_key: "8f1f1d1a067a94c07be83db82c547e35"
server_key_iv: "eaab9b2bf80abebf42131d8c2864608c"
server_key_version: "encryptionkey@0"
I20260929 20:53:29.092381 5526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "3882a14ace534ec2a9cd849da1d8d2af"
format_stamp: "Formatted at 2026-09-29 20:53:29 on dist-test-slave-m9g1"
server_key: "8f1f1d1a067a94c07be83db82c547e35"
server_key_iv: "eaab9b2bf80abebf42131d8c2864608c"
server_key_version: "encryptionkey@0"
I20260929 20:53:29.097388 5526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 20:53:29.100788 5540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:29.102131 5526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:53:29.102339 5526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "3882a14ace534ec2a9cd849da1d8d2af"
format_stamp: "Formatted at 2026-09-29 20:53:29 on dist-test-slave-m9g1"
server_key: "8f1f1d1a067a94c07be83db82c547e35"
server_key_iv: "eaab9b2bf80abebf42131d8c2864608c"
server_key_version: "encryptionkey@0"
I20260929 20:53:29.102531 5526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:29.118098 5526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:29.118927 5526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:29.119184 5526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:29.135821 5526 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40511
I20260929 20:53:29.135841 5591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40511 every 8 connection(s)
I20260929 20:53:29.137095 5526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:53:29.139077 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 5526
I20260929 20:53:29.139271 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:53:29.140000 388 external_mini_cluster.cc:1428] Setting key a53537302c50beea51c21792067e541f
I20260929 20:53:29.142511 5592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:29.152179 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:29.141630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40068 (local address 127.0.97.62:40511)
0929 20:53:29.142482 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:29.142493 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:29.142536 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:53:29.144079 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:29.144098 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:29.144527 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:29.144883 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:29.145399 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:29.146775 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:29.148064 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:29.148684 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:29.148843 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:29.149992 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:29.149999 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:29.150012 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:53:29.150634 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:29.150708 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:53:29.150728 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:29.150826 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:29.151034 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 20:53:29.151296 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":255,"threads_started":1}
I20260929 20:53:29.152542 5592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af: Bootstrap starting.
I20260929 20:53:29.155490 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:53:29.155515 5592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:29.156396 5592 log.cc:824] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:29.158978 5592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af: No bootstrap required, opened a new log
I20260929 20:53:29.165820 5592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3882a14ace534ec2a9cd849da1d8d2af" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40511 } }
I20260929 20:53:29.166175 5592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:29.166311 5592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3882a14ace534ec2a9cd849da1d8d2af, State: Initialized, Role: FOLLOWER
I20260929 20:53:29.166850 5592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3882a14ace534ec2a9cd849da1d8d2af" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40511 } }
I20260929 20:53:29.167059 5592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:53:29.167182 5592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:53:29.167338 5592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:29.169327 5592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3882a14ace534ec2a9cd849da1d8d2af" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40511 } }
I20260929 20:53:29.169790 5592 leader_election.cc:302] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3882a14ace534ec2a9cd849da1d8d2af; no voters:
I20260929 20:53:29.170342 5592 leader_election.cc:288] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:53:29.170473 5597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:29.171268 5597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af [term 1 LEADER]: Becoming Leader. State: Replica: 3882a14ace534ec2a9cd849da1d8d2af, State: Running, Role: LEADER
I20260929 20:53:29.171783 5597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3882a14ace534ec2a9cd849da1d8d2af" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40511 } }
I20260929 20:53:29.172421 5592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:53:29.174507 5599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3882a14ace534ec2a9cd849da1d8d2af. Latest consensus state: current_term: 1 leader_uuid: "3882a14ace534ec2a9cd849da1d8d2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3882a14ace534ec2a9cd849da1d8d2af" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40511 } } }
I20260929 20:53:29.174620 5598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3882a14ace534ec2a9cd849da1d8d2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3882a14ace534ec2a9cd849da1d8d2af" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40511 } } }
I20260929 20:53:29.174891 5599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:29.174923 5598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:29.176314 5605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:53:29.181830 5605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:53:29.190496 5605 catalog_manager.cc:1347] Generated new cluster ID: e04ef8dbe5584be2871cd51db9b2db68
I20260929 20:53:29.190606 5605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:53:29.214818 5605 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:53:29.216557 5605 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:53:29.227227 5605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3882a14ace534ec2a9cd849da1d8d2af: Generated new TSK 0
I20260929 20:53:29.228003 5605 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 20:53:29.274792 5622 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 20:53:29.274984 5622 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 20:53:29.275712 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:29.247673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40084 (local address 127.0.97.62:40511)
0929 20:53:29.247875 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:29.247882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:29.248302 (+ 420us) server_negotiation.cc:407] Connection header received
0929 20:53:29.248596 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:29.248599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:29.248661 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:29.248772 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:53:29.249398 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:29.250066 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:29.252083 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:29.252513 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:29.252598 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:29.253174 (+ 576us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:53:29.275167 (+ 21993us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 20:53:29.275484 (+ 317us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40084: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 20:53:29.275831 5595 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40084: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 20:53:29.277526 5621 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:29.247565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40511 (local address 127.0.0.1:40084)
0929 20:53:29.248198 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:29.248221 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:29.248397 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:29.248826 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:29.248831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:29.248844 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:53:29.249246 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:29.249257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:29.250216 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:29.250222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:29.251958 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:29.251973 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:29.252619 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:29.252679 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:29.252855 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:29.252938 (+ 83us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:53:29.275462 (+ 22524us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 20:53:29.275807 (+ 345us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:40511: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":300,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:40511: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 20:53:29.279554 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 5526
I20260929 20:53:29.280697 5587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:29.371016 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 5526
2026-09-29T20:53:29Z chronyd exiting
I20260929 20:53:29.417567 388 test_util.cc:175] -----------------------------------------------
I20260929 20:53:29.417714 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715169435170-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (704 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 20:53:29.421275 388 mini_oidc.cc:152] Starting JWKS server
I20260929 20:53:29.423200 388 webserver.cc:460] Webserver started at https://127.0.0.1:42177/ using document root <none> and password file <none>
I20260929 20:53:29.423410 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 20:53:29.424029 388 webserver.cc:460] Webserver started at http://127.0.0.1:43391/ using document root <none> and password file <none>
2026-09-29T20:53:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:53:29Z Disabled control of system clock
I20260929 20:53:29.447156 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39069
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39069
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42177/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715169435170-388-0/testchainca.cert with env {}
W20260929 20:53:29.749519 5629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:29.749810 5629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:29.749928 5629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:29.756990 5629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:53:29.757090 5629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:29.757128 5629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:29.757208 5629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:53:29.757261 5629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:53:29.757349 5629 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:53:29.757395 5629 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42177/jwks.json
I20260929 20:53:29.767056 5629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39069
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42177/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715169435170-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:29.768332 5629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:29.769557 5629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:29.780571 5634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:29.780584 5635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:29.780560 5637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:29.781217 5629 server_base.cc:1034] running on GCE node
I20260929 20:53:29.781948 5629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:29.783290 5629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:29.784482 5629 hybrid_clock.cc:648] HybridClock initialized: now 1790715209784434 us; error 43 us; skew 500 ppm
I20260929 20:53:29.784814 5629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:53:29.786945 5629 webserver.cc:460] Webserver started at http://127.0.97.62:35409/ using document root <none> and password file <none>
I20260929 20:53:29.787593 5629 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:29.787703 5629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:29.787992 5629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:29.790791 5629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "17f07468f3db47ba947ba4f6292049b9"
format_stamp: "Formatted at 2026-09-29 20:53:29 on dist-test-slave-m9g1"
server_key: "bee3320cecad38139ae21b387490e00b"
server_key_iv: "6e2f28391c148dce2c8f808a0043ca9a"
server_key_version: "encryptionkey@0"
I20260929 20:53:29.791558 5629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "17f07468f3db47ba947ba4f6292049b9"
format_stamp: "Formatted at 2026-09-29 20:53:29 on dist-test-slave-m9g1"
server_key: "bee3320cecad38139ae21b387490e00b"
server_key_iv: "6e2f28391c148dce2c8f808a0043ca9a"
server_key_version: "encryptionkey@0"
I20260929 20:53:29.796408 5629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:53:29.799695 5643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:29.801033 5629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:53:29.801195 5629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "17f07468f3db47ba947ba4f6292049b9"
format_stamp: "Formatted at 2026-09-29 20:53:29 on dist-test-slave-m9g1"
server_key: "bee3320cecad38139ae21b387490e00b"
server_key_iv: "6e2f28391c148dce2c8f808a0043ca9a"
server_key_version: "encryptionkey@0"
I20260929 20:53:29.801388 5629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:29.818449 5629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:29.819226 5629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:29.819463 5629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:29.835429 5629 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39069
I20260929 20:53:29.835448 5694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39069 every 8 connection(s)
I20260929 20:53:29.836618 5629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:53:29.840133 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:29.842635 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 5629
I20260929 20:53:29.842777 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:53:29.843339 388 external_mini_cluster.cc:1428] Setting key 94c91826c6871239b0c831125ebaca21
I20260929 20:53:29.850631 5695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9: Bootstrap starting.
I20260929 20:53:29.853425 5695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:29.854463 5695 log.cc:824] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:29.855620 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:29.845098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.0.97.62:39069)
0929 20:53:29.845953 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:29.845970 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:29.846008 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:53:29.847570 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:29.847593 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:29.848043 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:29.848384 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:29.848931 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:29.850115 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:29.851378 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:29.852037 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:29.852196 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:29.853206 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:29.853213 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:29.853227 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:53:29.853778 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:29.853871 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:53:29.853891 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:29.853984 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:29.854221 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 20:53:29.854491 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":276,"threads_started":1}
I20260929 20:53:29.857292 5695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9: No bootstrap required, opened a new log
I20260929 20:53:29.859030 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:53:29.864461 5695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f07468f3db47ba947ba4f6292049b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39069 } }
I20260929 20:53:29.864835 5695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:29.864926 5695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f07468f3db47ba947ba4f6292049b9, State: Initialized, Role: FOLLOWER
I20260929 20:53:29.865514 5695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f07468f3db47ba947ba4f6292049b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39069 } }
I20260929 20:53:29.865742 5695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:53:29.865877 5695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:53:29.866057 5695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:29.868880 5695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f07468f3db47ba947ba4f6292049b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39069 } }
I20260929 20:53:29.869470 5695 leader_election.cc:302] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17f07468f3db47ba947ba4f6292049b9; no voters:
I20260929 20:53:29.870007 5695 leader_election.cc:288] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:53:29.870163 5700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:29.870826 5700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9 [term 1 LEADER]: Becoming Leader. State: Replica: 17f07468f3db47ba947ba4f6292049b9, State: Running, Role: LEADER
I20260929 20:53:29.871400 5700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f07468f3db47ba947ba4f6292049b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39069 } }
I20260929 20:53:29.872097 5695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:53:29.874256 5702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17f07468f3db47ba947ba4f6292049b9. Latest consensus state: current_term: 1 leader_uuid: "17f07468f3db47ba947ba4f6292049b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f07468f3db47ba947ba4f6292049b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39069 } } }
I20260929 20:53:29.874617 5702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:29.875083 5701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17f07468f3db47ba947ba4f6292049b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f07468f3db47ba947ba4f6292049b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39069 } } }
I20260929 20:53:29.875748 5701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:29.882906 5707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:53:29.887679 5707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:53:29.895154 5707 catalog_manager.cc:1347] Generated new cluster ID: 7e75fe8751b3433a9a0379e599dd150a
I20260929 20:53:29.895262 5707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:53:29.933801 5707 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:53:29.935091 5707 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:53:29.943199 5707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17f07468f3db47ba947ba4f6292049b9: Generated new TSK 0
I20260929 20:53:29.943956 5707 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 20:53:29.986913 5725 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 20:53:29.987046 5725 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 20:53:29.988016 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:29.961835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53408 (local address 127.0.97.62:39069)
0929 20:53:29.961975 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:29.961981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:29.962400 (+ 419us) server_negotiation.cc:407] Connection header received
0929 20:53:29.962616 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:29.962620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:29.962690 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:29.962815 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:53:29.963248 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:29.963937 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:29.965312 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:29.965754 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:29.965858 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:29.966342 (+ 484us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:53:29.987368 (+ 21026us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 20:53:29.987741 (+ 373us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53408: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 20:53:29.988152 5698 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53408: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 20:53:29.988248 5724 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:29.961772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39069 (local address 127.0.0.1:53408)
0929 20:53:29.962311 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:29.962325 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:29.962452 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:29.962847 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:29.962852 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:29.962864 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:53:29.963116 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:29.963126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:29.964070 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:29.964076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:29.965176 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:29.965188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:29.965865 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:29.965911 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:29.966055 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:29.966088 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:53:29.987605 (+ 21517us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 20:53:29.987872 (+ 267us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39069: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":291,"threads_started":1}
I20260929 20:53:29.989771 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 5629
I20260929 20:53:29.991004 5690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:30.083593 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 5629
2026-09-29T20:53:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (714 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 20:53:30.136303 388 mini_oidc.cc:152] Starting JWKS server
I20260929 20:53:30.138063 388 webserver.cc:460] Webserver started at https://127.0.0.1:41525/ using document root <none> and password file <none>
I20260929 20:53:30.138309 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 20:53:30.138849 388 webserver.cc:460] Webserver started at http://127.0.0.1:39313/ using document root <none> and password file <none>
2026-09-29T20:53:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:53:30Z Disabled control of system clock
I20260929 20:53:30.160655 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43289
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43289
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41525/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 20:53:30.459609 5732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:30.459924 5732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:30.460067 5732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:30.467097 5732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:53:30.467207 5732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:30.467270 5732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:30.467350 5732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:53:30.467409 5732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:53:30.467494 5732 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:53:30.467546 5732 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41525/jwks.json
I20260929 20:53:30.476430 5732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43289
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41525/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:30.477681 5732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:30.479439 5732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:30.490222 5740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:30.490357 5737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:30.490317 5738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:30.491235 5732 server_base.cc:1034] running on GCE node
I20260929 20:53:30.491983 5732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:30.493285 5732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:30.494472 5732 hybrid_clock.cc:648] HybridClock initialized: now 1790715210494429 us; error 37 us; skew 500 ppm
I20260929 20:53:30.494793 5732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:53:30.501890 5732 webserver.cc:460] Webserver started at http://127.0.97.62:33549/ using document root <none> and password file <none>
I20260929 20:53:30.502599 5732 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:30.502691 5732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:30.502995 5732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:30.505757 5732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "31f444860b69440d8faa76d26a328785"
format_stamp: "Formatted at 2026-09-29 20:53:30 on dist-test-slave-m9g1"
server_key: "2ebbfe08729549b69b71486441a8c787"
server_key_iv: "b290ed86d4d203de062857abdf192605"
server_key_version: "encryptionkey@0"
I20260929 20:53:30.506518 5732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "31f444860b69440d8faa76d26a328785"
format_stamp: "Formatted at 2026-09-29 20:53:30 on dist-test-slave-m9g1"
server_key: "2ebbfe08729549b69b71486441a8c787"
server_key_iv: "b290ed86d4d203de062857abdf192605"
server_key_version: "encryptionkey@0"
I20260929 20:53:30.511149 5732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 20:53:30.514318 5746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:30.515554 5732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:53:30.515698 5732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "31f444860b69440d8faa76d26a328785"
format_stamp: "Formatted at 2026-09-29 20:53:30 on dist-test-slave-m9g1"
server_key: "2ebbfe08729549b69b71486441a8c787"
server_key_iv: "b290ed86d4d203de062857abdf192605"
server_key_version: "encryptionkey@0"
I20260929 20:53:30.515870 5732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:30.535266 5732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:30.536056 5732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:30.536299 5732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:30.552729 5732 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43289
I20260929 20:53:30.552749 5797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43289 every 8 connection(s)
I20260929 20:53:30.553984 5732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:53:30.554579 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 5732
I20260929 20:53:30.554757 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:53:30.555466 388 external_mini_cluster.cc:1428] Setting key 0491d42258bf639cb15b624e6b82edad
I20260929 20:53:30.558127 5798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:30.567677 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:30.557235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51190 (local address 127.0.97.62:43289)
0929 20:53:30.558175 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:30.558212 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:30.558258 (+ 46us) server_negotiation.cc:407] Connection header received
0929 20:53:30.559749 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:30.559770 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:30.560196 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:30.560552 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:30.561082 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:30.562383 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:30.563633 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:30.564252 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:30.564408 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:30.565489 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:30.565496 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:30.565509 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:53:30.566061 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:30.566133 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:53:30.566152 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:30.566275 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:30.566539 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 20:53:30.566805 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":335,"threads_started":1}
I20260929 20:53:30.568372 5798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785: Bootstrap starting.
I20260929 20:53:30.571581 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:53:30.571823 5798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:30.572961 5798 log.cc:824] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:30.576516 5798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785: No bootstrap required, opened a new log
I20260929 20:53:30.584211 5798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f444860b69440d8faa76d26a328785" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43289 } }
I20260929 20:53:30.584560 5798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:30.584656 5798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f444860b69440d8faa76d26a328785, State: Initialized, Role: FOLLOWER
I20260929 20:53:30.585176 5798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f444860b69440d8faa76d26a328785" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43289 } }
I20260929 20:53:30.585376 5798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:53:30.585495 5798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:53:30.585645 5798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:30.587690 5798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f444860b69440d8faa76d26a328785" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43289 } }
I20260929 20:53:30.588145 5798 leader_election.cc:302] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31f444860b69440d8faa76d26a328785; no voters:
I20260929 20:53:30.588626 5798 leader_election.cc:288] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:53:30.588738 5803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:30.589331 5803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785 [term 1 LEADER]: Becoming Leader. State: Replica: 31f444860b69440d8faa76d26a328785, State: Running, Role: LEADER
I20260929 20:53:30.589789 5803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f444860b69440d8faa76d26a328785" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43289 } }
I20260929 20:53:30.590615 5798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:53:30.592567 5805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31f444860b69440d8faa76d26a328785. Latest consensus state: current_term: 1 leader_uuid: "31f444860b69440d8faa76d26a328785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f444860b69440d8faa76d26a328785" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43289 } } }
I20260929 20:53:30.592592 5804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31f444860b69440d8faa76d26a328785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f444860b69440d8faa76d26a328785" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43289 } } }
I20260929 20:53:30.592988 5804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:30.592988 5805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:30.594439 5810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:53:30.599229 5810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:53:30.607656 5810 catalog_manager.cc:1347] Generated new cluster ID: d78940099bc843b18f1cff3731441f59
I20260929 20:53:30.607755 5810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:53:30.615487 5810 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:53:30.616626 5810 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:53:30.632879 5810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31f444860b69440d8faa76d26a328785: Generated new TSK 0
I20260929 20:53:30.634071 5810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:53:30.661199 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:30.653509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51202 (local address 127.0.97.62:43289)
0929 20:53:30.653692 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:30.653698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:30.654245 (+ 547us) server_negotiation.cc:407] Connection header received
0929 20:53:30.654491 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:30.654495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:30.654557 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:30.654674 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:53:30.655165 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:30.655839 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:30.657756 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:30.658207 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:30.658298 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:30.658871 (+ 573us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:53:30.660596 (+ 1725us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41525/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 20:53:30.660887 (+ 291us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51202: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41525/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 20:53:30.661423 5827 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:30.653510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43289 (local address 127.0.0.1:51202)
0929 20:53:30.654127 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:30.654147 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:30.654328 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:30.654711 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:30.654716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:30.654729 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:53:30.655034 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:30.655046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:30.655975 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:30.655980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:30.657625 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:30.657641 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:30.658328 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:30.658397 (+ 69us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:30.658588 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:30.658633 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:53:30.660776 (+ 2143us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41525/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 20:53:30.661022 (+ 246us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:43289: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41525/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":305,"threads_started":1}
W20260929 20:53:30.661756 5801 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51202: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41525/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260929 20:53:30.662591 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 5732
I20260929 20:53:30.663569 5793 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 20:53:30.673031 5828 webserver.cc:533] Webserver: sslize failed:
E20260929 20:53:30.673139 5828 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 20:53:30.751802 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 5732
2026-09-29T20:53:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (666 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T20:53:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:53:30Z Disabled control of system clock
I20260929 20:53:30.824921 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40937
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40937
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715169435170-388-0/testchainca.cert with env {}
W20260929 20:53:31.127097 5833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:31.127380 5833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:31.127475 5833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:31.134552 5833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:53:31.134686 5833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:31.134752 5833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:31.134829 5833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:53:31.134888 5833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:53:31.134969 5833 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:53:31.135022 5833 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 20:53:31.144589 5833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40937
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715169435170-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:31.145792 5833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:31.147044 5833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:31.157871 5838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:31.157904 5839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:31.157933 5841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:31.158629 5833 server_base.cc:1034] running on GCE node
I20260929 20:53:31.159358 5833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:31.160660 5833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:31.161859 5833 hybrid_clock.cc:648] HybridClock initialized: now 1790715211161802 us; error 46 us; skew 500 ppm
I20260929 20:53:31.162176 5833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:53:31.164306 5833 webserver.cc:460] Webserver started at http://127.0.97.62:41277/ using document root <none> and password file <none>
I20260929 20:53:31.164955 5833 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:31.165045 5833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:31.165333 5833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:31.168089 5833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "ade949163a4f4d3b8da316d6abe82945"
format_stamp: "Formatted at 2026-09-29 20:53:31 on dist-test-slave-m9g1"
server_key: "f5cd29a867726b71ff6c12df017eb9d9"
server_key_iv: "93c691704f28f17047dcf15d737183b6"
server_key_version: "encryptionkey@0"
I20260929 20:53:31.168831 5833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "ade949163a4f4d3b8da316d6abe82945"
format_stamp: "Formatted at 2026-09-29 20:53:31 on dist-test-slave-m9g1"
server_key: "f5cd29a867726b71ff6c12df017eb9d9"
server_key_iv: "93c691704f28f17047dcf15d737183b6"
server_key_version: "encryptionkey@0"
I20260929 20:53:31.173740 5833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:53:31.176924 5847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:31.178164 5833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 20:53:31.178359 5833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "ade949163a4f4d3b8da316d6abe82945"
format_stamp: "Formatted at 2026-09-29 20:53:31 on dist-test-slave-m9g1"
server_key: "f5cd29a867726b71ff6c12df017eb9d9"
server_key_iv: "93c691704f28f17047dcf15d737183b6"
server_key_version: "encryptionkey@0"
I20260929 20:53:31.178536 5833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:31.207557 5833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:31.208412 5833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:31.208683 5833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:31.224972 5833 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40937
I20260929 20:53:31.224993 5898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40937 every 8 connection(s)
I20260929 20:53:31.226249 5833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:53:31.229848 5899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:31.229949 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 5833
I20260929 20:53:31.230126 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:53:31.230756 388 external_mini_cluster.cc:1428] Setting key dfe703824d58415bd54638f52b5493f3
I20260929 20:53:31.241254 5899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945: Bootstrap starting.
I20260929 20:53:31.243551 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:31.232531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50490 (local address 127.0.97.62:40937)
0929 20:53:31.233431 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:31.233441 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:31.233484 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:53:31.235545 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:31.235567 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:31.236039 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:31.236398 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:31.236903 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:31.238173 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:31.239418 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:31.240081 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:31.240239 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:31.241339 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:31.241346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:31.241360 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:53:31.241902 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:31.241980 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:53:31.242002 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:31.242095 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:31.242345 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 20:53:31.242655 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":278,"threads_started":1}
I20260929 20:53:31.244486 5899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:31.245527 5899 log.cc:824] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:31.247180 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:53:31.248515 5899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945: No bootstrap required, opened a new log
I20260929 20:53:31.255925 5899 raft_consensus.cc:348] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade949163a4f4d3b8da316d6abe82945" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40937 } }
I20260929 20:53:31.256242 5899 raft_consensus.cc:374] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:31.256309 5899 raft_consensus.cc:729] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ade949163a4f4d3b8da316d6abe82945, State: Initialized, Role: FOLLOWER
I20260929 20:53:31.256786 5899 consensus_queue.cc:260] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade949163a4f4d3b8da316d6abe82945" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40937 } }
I20260929 20:53:31.256980 5899 raft_consensus.cc:388] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:53:31.257069 5899 raft_consensus.cc:482] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:53:31.257179 5899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:31.259222 5899 raft_consensus.cc:504] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade949163a4f4d3b8da316d6abe82945" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40937 } }
I20260929 20:53:31.259791 5899 leader_election.cc:302] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ade949163a4f4d3b8da316d6abe82945; no voters:
I20260929 20:53:31.260303 5899 leader_election.cc:288] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:53:31.260416 5904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:31.260995 5904 raft_consensus.cc:686] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945 [term 1 LEADER]: Becoming Leader. State: Replica: ade949163a4f4d3b8da316d6abe82945, State: Running, Role: LEADER
I20260929 20:53:31.261507 5904 consensus_queue.cc:237] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade949163a4f4d3b8da316d6abe82945" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40937 } }
I20260929 20:53:31.262154 5899 sys_catalog.cc:564] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:53:31.263999 5905 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ade949163a4f4d3b8da316d6abe82945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade949163a4f4d3b8da316d6abe82945" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40937 } } }
I20260929 20:53:31.264416 5905 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:31.265043 5906 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ade949163a4f4d3b8da316d6abe82945. Latest consensus state: current_term: 1 leader_uuid: "ade949163a4f4d3b8da316d6abe82945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade949163a4f4d3b8da316d6abe82945" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40937 } } }
I20260929 20:53:31.265363 5906 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:31.265666 5913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:53:31.272701 5913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:53:31.280447 5913 catalog_manager.cc:1347] Generated new cluster ID: 1adffdda193f45b2b463d07ab989153c
I20260929 20:53:31.280542 5913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:53:31.301087 5913 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:53:31.302359 5913 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:53:31.314571 5913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ade949163a4f4d3b8da316d6abe82945: Generated new TSK 0
I20260929 20:53:31.315387 5913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:53:31.368244 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:31.337581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.0.97.62:40937)
0929 20:53:31.337785 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:31.337792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:31.338248 (+ 456us) server_negotiation.cc:407] Connection header received
0929 20:53:31.338495 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:31.338499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:31.338573 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:31.338712 (+ 139us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:53:31.339184 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:31.339915 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:31.341900 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:31.342351 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:31.342445 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:31.342962 (+ 517us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:53:31.367731 (+ 24769us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 20:53:31.368020 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50500: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 20:53:31.368377 5902 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50500: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 20:53:31.368978 5928 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:31.337552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40937 (local address 127.0.0.1:50500)
0929 20:53:31.338124 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:31.338143 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:31.338324 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:31.338740 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:31.338746 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:31.338758 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:53:31.339048 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:31.339060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:31.340041 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:31.340047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:31.341775 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:31.341789 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:31.342450 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:31.342504 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:31.342693 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:31.342738 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:53:31.367987 (+ 25249us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 20:53:31.368277 (+ 290us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:40937: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":281,"threads_started":1}
I20260929 20:53:31.370065 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 5833
I20260929 20:53:31.371325 5894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:31.463081 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 5833
2026-09-29T20:53:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (690 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (379 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (383 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:53:32 dist-test-slave-m9g1 krb5kdc[5943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:53:32 dist-test-slave-m9g1 krb5kdc[5943](info): set up 2 sockets
Sep 29 20:53:32 dist-test-slave-m9g1 krb5kdc[5943](info): commencing operation
krb5kdc: starting...
W20260929 20:53:33.210510 388 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:53:33 dist-test-slave-m9g1 krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:53:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:53:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:53:33.384905 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45313
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:33.686203 5956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:33.686506 5956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:33.686635 5956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:33.693728 5956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:53:33.693846 5956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:33.693912 5956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:33.693984 5956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:53:33.694042 5956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:53:33.703598 5956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:33.704816 5956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:33.706068 5956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:33.716843 5962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:33.716882 5964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:33.717470 5956 server_base.cc:1034] running on GCE node
W20260929 20:53:33.717473 5961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:33.718277 5956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:33.719528 5956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:33.720706 5956 hybrid_clock.cc:648] HybridClock initialized: now 1790715213720656 us; error 49 us; skew 500 ppm
I20260929 20:53:33.720991 5956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:33 dist-test-slave-m9g1 krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:33.723737 5956 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:53:33.724964 5956 webserver.cc:460] Webserver started at http://127.0.97.62:41815/ using document root <none> and password file <none>
I20260929 20:53:33.725548 5956 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:33.725649 5956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:33.725924 5956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:33.728654 5956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "57c1a2ee276044688ec545e473d89eb4"
format_stamp: "Formatted at 2026-09-29 20:53:33 on dist-test-slave-m9g1"
server_key: "ebf57118f5683866e141501793775f58"
server_key_iv: "542cee4befc4773e4b2b86c79a3d56d5"
server_key_version: "encryptionkey@0"
I20260929 20:53:33.729357 5956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "57c1a2ee276044688ec545e473d89eb4"
format_stamp: "Formatted at 2026-09-29 20:53:33 on dist-test-slave-m9g1"
server_key: "ebf57118f5683866e141501793775f58"
server_key_iv: "542cee4befc4773e4b2b86c79a3d56d5"
server_key_version: "encryptionkey@0"
I20260929 20:53:33.733898 5956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 20:53:33.737144 5971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:33.738384 5956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:53:33.738510 5956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "57c1a2ee276044688ec545e473d89eb4"
format_stamp: "Formatted at 2026-09-29 20:53:33 on dist-test-slave-m9g1"
server_key: "ebf57118f5683866e141501793775f58"
server_key_iv: "542cee4befc4773e4b2b86c79a3d56d5"
server_key_version: "encryptionkey@0"
I20260929 20:53:33.738675 5956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:33.757717 5956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:33.761483 5956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:33.761703 5956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:33.777817 5956 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45313
I20260929 20:53:33.777863 6022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45313 every 8 connection(s)
I20260929 20:53:33.779181 5956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:53:33.780135 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 5956
I20260929 20:53:33.780326 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:53:33.780948 388 external_mini_cluster.cc:1428] Setting key c1df5b32df42124ccb6b7a3db95d7572
I20260929 20:53:33.783455 6023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:53:33 dist-test-slave-m9g1 krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:33.795956 6023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4: Bootstrap starting.
I20260929 20:53:33.799156 6023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:33.800163 6023 log.cc:824] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:33.802772 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:33.782805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58744 (local address 127.0.97.62:45313)
0929 20:53:33.783522 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:33.783537 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:33.783585 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:53:33.785143 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:33.785165 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:33.785535 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:33.785875 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:33.787567 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:33.788909 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:33.790658 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:33.791321 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:33.791441 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:33.795841 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:33.795849 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:33.795866 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:33.795908 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:33.798838 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:33.799569 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:33.799576 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:33.799584 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:33.799755 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:33.800114 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:33.800118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:33.800120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:33.800700 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:33.801027 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:33.801365 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 20:53:33.801760 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":323,"threads_started":1}
I20260929 20:53:33.803086 6023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4: No bootstrap required, opened a new log
I20260929 20:53:33.809994 6023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c1a2ee276044688ec545e473d89eb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45313 } }
I20260929 20:53:33.810390 6023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:33.810475 6023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57c1a2ee276044688ec545e473d89eb4, State: Initialized, Role: FOLLOWER
I20260929 20:53:33.811017 6023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c1a2ee276044688ec545e473d89eb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45313 } }
I20260929 20:53:33.811224 6023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:53:33.811344 6023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:53:33.811482 6023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:33.813519 6023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c1a2ee276044688ec545e473d89eb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45313 } }
I20260929 20:53:33.813966 6023 leader_election.cc:302] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57c1a2ee276044688ec545e473d89eb4; no voters:
I20260929 20:53:33.814512 6023 leader_election.cc:288] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:53:33.814625 6028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:33.815214 6028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4 [term 1 LEADER]: Becoming Leader. State: Replica: 57c1a2ee276044688ec545e473d89eb4, State: Running, Role: LEADER
I20260929 20:53:33.815686 6028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c1a2ee276044688ec545e473d89eb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45313 } }
I20260929 20:53:33.816466 6023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:53:33.818370 6029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57c1a2ee276044688ec545e473d89eb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c1a2ee276044688ec545e473d89eb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45313 } } }
I20260929 20:53:33.818755 6029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:33.818636 6030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57c1a2ee276044688ec545e473d89eb4. Latest consensus state: current_term: 1 leader_uuid: "57c1a2ee276044688ec545e473d89eb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c1a2ee276044688ec545e473d89eb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45313 } } }
I20260929 20:53:33.818867 6030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:33.820060 6036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:53:33.824888 6036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:53:33.833227 6036 catalog_manager.cc:1347] Generated new cluster ID: adb81879b1a84003b167e3b5561b2cbd
I20260929 20:53:33.833333 6036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:53:33.842857 6036 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:53:33.844300 6036 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:53:33.851750 6036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57c1a2ee276044688ec545e473d89eb4: Generated new TSK 0
I20260929 20:53:33.852732 6036 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:53:33.914898 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45313
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:34.244755 6049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:34.245023 6049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:34.245117 6049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:34.252220 6049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:34.252327 6049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:34.252447 6049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:53:34.261754 6049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:34.262970 6049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:34.264196 6049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:34.275998 6054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:34.276081 6057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:34.276055 6055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:34.276895 6049 server_base.cc:1034] running on GCE node
I20260929 20:53:34.277415 6049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:34.278260 6049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:34.279461 6049 hybrid_clock.cc:648] HybridClock initialized: now 1790715214279412 us; error 45 us; skew 500 ppm
I20260929 20:53:34.279734 6049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:34 dist-test-slave-m9g1 krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:34.282683 6049 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:53:34.283990 6049 webserver.cc:460] Webserver started at http://127.0.97.1:41217/ using document root <none> and password file <none>
I20260929 20:53:34.284590 6049 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:34.284682 6049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:34.284920 6049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:34.287652 6049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "626c7eec2cf044daa2865f79865833dc"
format_stamp: "Formatted at 2026-09-29 20:53:34 on dist-test-slave-m9g1"
server_key: "ae2080c03ed58133dfec7ccaaffe29ff"
server_key_iv: "b7b050748a4b75d3f6635e0e813f1478"
server_key_version: "encryptionkey@0"
I20260929 20:53:34.288337 6049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "626c7eec2cf044daa2865f79865833dc"
format_stamp: "Formatted at 2026-09-29 20:53:34 on dist-test-slave-m9g1"
server_key: "ae2080c03ed58133dfec7ccaaffe29ff"
server_key_iv: "b7b050748a4b75d3f6635e0e813f1478"
server_key_version: "encryptionkey@0"
I20260929 20:53:34.293058 6049 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:53:34.296382 6064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:34.297632 6049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:53:34.297760 6049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "626c7eec2cf044daa2865f79865833dc"
format_stamp: "Formatted at 2026-09-29 20:53:34 on dist-test-slave-m9g1"
server_key: "ae2080c03ed58133dfec7ccaaffe29ff"
server_key_iv: "b7b050748a4b75d3f6635e0e813f1478"
server_key_version: "encryptionkey@0"
I20260929 20:53:34.297945 6049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:34.313985 6049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:34.317761 6049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:34.318001 6049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:34.318923 6049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:34.320147 6049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:34.320218 6049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:34.320331 6049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:34.320385 6049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:34.348063 6049 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38769
I20260929 20:53:34.348095 6176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38769 every 8 connection(s)
I20260929 20:53:34.349215 6049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:53:34.354998 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 6049
I20260929 20:53:34.355201 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:53:34.355942 388 external_mini_cluster.cc:1428] Setting key 840aaaea14ffab19f5c656e085d403d5
Sep 29 20:53:34 dist-test-slave-m9g1 krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:34.369313 6179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:34.351515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40845 (local address 127.0.97.62:45313)
0929 20:53:34.351902 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:34.351912 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:34.352841 (+ 929us) server_negotiation.cc:407] Connection header received
0929 20:53:34.354610 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:34.354615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:34.354677 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:34.354768 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:34.356660 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:34.357618 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:34.359766 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:34.360142 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:34.360229 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:34.363823 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:34.363828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:34.363831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:34.363859 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:34.366716 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:34.367343 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:34.367348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:34.367352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:34.367484 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:34.367813 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:34.367818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:34.367821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:34.368124 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:34.368253 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:34.368813 (+ 560us) server_negotiation.cc:299] Negotiation successful
0929 20:53:34.368999 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":171,"threads_started":1}
I20260929 20:53:34.370092 6180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:34.352021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45313 (local address 127.0.97.1:40845)
0929 20:53:34.352693 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:34.352735 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:34.354353 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:34.355041 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:34.355052 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:34.355734 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:34.356496 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:34.356523 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:34.357765 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:34.357769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:34.359641 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:34.359656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:34.360326 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:34.360332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:34.360573 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:34.361191 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:34.361215 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:34.363627 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:34.366841 (+ 3214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:34.366848 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:34.366859 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:34.367233 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:34.367567 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:34.367569 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:34.367572 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:34.367727 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:34.368223 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:34.368228 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:34.368503 (+ 275us) client_negotiation.cc:769] Sending connection context
0929 20:53:34.368705 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 20:53:34.369056 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":262,"threads_started":1}
I20260929 20:53:34.371439 6177 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45313
I20260929 20:53:34.371745 6177 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:34.372603 6177 heartbeater.cc:511] Master 127.0.97.62:45313 requested a full tablet report, sending...
I20260929 20:53:34.374708 5988 ts_manager.cc:194] Registered new tserver with Master: 626c7eec2cf044daa2865f79865833dc (127.0.97.1:38769)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:53:34.376094 5988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:53:34.398048 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45313
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:34.698698 6183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:34.698978 6183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:34.699075 6183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:34.706216 6183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:34.706321 6183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:34.706436 6183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:53:34.715828 6183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:34.717065 6183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:34.718350 6183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:34.730101 6188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:34.730226 6191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:34.730259 6189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:34.730728 6183 server_base.cc:1034] running on GCE node
I20260929 20:53:34.731271 6183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:34.732086 6183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:34.733263 6183 hybrid_clock.cc:648] HybridClock initialized: now 1790715214733209 us; error 51 us; skew 500 ppm
I20260929 20:53:34.733522 6183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:34 dist-test-slave-m9g1 krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:34.736399 6183 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:53:34.737607 6183 webserver.cc:460] Webserver started at http://127.0.97.2:33853/ using document root <none> and password file <none>
I20260929 20:53:34.738272 6183 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:34.738384 6183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:34.738636 6183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:34.741276 6183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "7164168cbfd54a12aa0a2226f802bb9b"
format_stamp: "Formatted at 2026-09-29 20:53:34 on dist-test-slave-m9g1"
server_key: "68367ef743270083d82bba6fa99769d6"
server_key_iv: "12609e9f94b18906b8495ba93ba36027"
server_key_version: "encryptionkey@0"
I20260929 20:53:34.742049 6183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7164168cbfd54a12aa0a2226f802bb9b"
format_stamp: "Formatted at 2026-09-29 20:53:34 on dist-test-slave-m9g1"
server_key: "68367ef743270083d82bba6fa99769d6"
server_key_iv: "12609e9f94b18906b8495ba93ba36027"
server_key_version: "encryptionkey@0"
I20260929 20:53:34.746611 6183 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:53:34.749698 6198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:34.750926 6183 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 20:53:34.751065 6183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "7164168cbfd54a12aa0a2226f802bb9b"
format_stamp: "Formatted at 2026-09-29 20:53:34 on dist-test-slave-m9g1"
server_key: "68367ef743270083d82bba6fa99769d6"
server_key_iv: "12609e9f94b18906b8495ba93ba36027"
server_key_version: "encryptionkey@0"
I20260929 20:53:34.751227 6183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:34.764430 6183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:34.768309 6183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:34.768553 6183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:34.769428 6183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:34.770695 6183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:34.770767 6183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:34.770890 6183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:34.770953 6183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:34.798972 6183 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38347
I20260929 20:53:34.798974 6310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38347 every 8 connection(s)
I20260929 20:53:34.800163 6183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:53:34.803774 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 6183
I20260929 20:53:34.803946 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:53:34.804476 388 external_mini_cluster.cc:1428] Setting key 421c54dd690d2aa9f201904583bd43fc
Sep 29 20:53:34 dist-test-slave-m9g1 krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:34.820494 6179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:34.802526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60037 (local address 127.0.97.62:45313)
0929 20:53:34.802683 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:34.802687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:34.803897 (+ 1210us) server_negotiation.cc:407] Connection header received
0929 20:53:34.806035 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:34.806039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:34.806106 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:34.806221 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:34.808235 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:34.808912 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:34.810885 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:34.811285 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:34.811372 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:34.814876 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:34.814883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:34.814887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:34.814923 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:34.817719 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:34.818413 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:34.818419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:34.818422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:34.818565 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:34.818920 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:34.818924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:34.818927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:34.819223 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:34.819367 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:34.819962 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 20:53:34.820165 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 20:53:34.821262 6313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:34.802947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45313 (local address 127.0.97.2:60037)
0929 20:53:34.803732 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:34.803790 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:34.805778 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:34.806695 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:34.806706 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:34.807357 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:34.808092 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:34.808114 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:34.809044 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:34.809050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:34.810765 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:34.810779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:34.811476 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:34.811482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:34.811756 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:34.812400 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:34.812426 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:34.814643 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:34.817852 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:34.817858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:34.817871 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:34.818308 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:34.818651 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:34.818654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:34.818656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:34.818826 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:34.819316 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:34.819322 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:34.819607 (+ 285us) client_negotiation.cc:769] Sending connection context
0929 20:53:34.819820 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 20:53:34.820190 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":271,"threads_started":1}
I20260929 20:53:34.822610 6311 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45313
I20260929 20:53:34.822954 6311 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:34.823864 6311 heartbeater.cc:511] Master 127.0.97.62:45313 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:53:34.825363 5988 ts_manager.cc:194] Registered new tserver with Master: 7164168cbfd54a12aa0a2226f802bb9b (127.0.97.2:38347)
I20260929 20:53:34.826169 5988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:53:34.846796 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45313
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:35.146977 6316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:35.147254 6316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:35.147377 6316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:35.154349 6316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:35.154465 6316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:35.154613 6316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:53:35.163996 6316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:35.165169 6316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:35.166389 6316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:35.178253 6321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:35.178664 6322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:35.178692 6324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:35.179215 6316 server_base.cc:1034] running on GCE node
I20260929 20:53:35.179759 6316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:35.180608 6316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:35.181818 6316 hybrid_clock.cc:648] HybridClock initialized: now 1790715215181772 us; error 35 us; skew 500 ppm
I20260929 20:53:35.182107 6316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:35 dist-test-slave-m9g1 krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:35.184899 6316 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:53:35.186162 6316 webserver.cc:460] Webserver started at http://127.0.97.3:33167/ using document root <none> and password file <none>
I20260929 20:53:35.186830 6316 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:35.186939 6316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:35.187209 6316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:35.190022 6316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "101e97da2b5e49b8a64585e549180afd"
format_stamp: "Formatted at 2026-09-29 20:53:35 on dist-test-slave-m9g1"
server_key: "c9135842b5c56460367d375373461902"
server_key_iv: "5608fcad0ba45178f64fe01b3ee37515"
server_key_version: "encryptionkey@0"
I20260929 20:53:35.190789 6316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "101e97da2b5e49b8a64585e549180afd"
format_stamp: "Formatted at 2026-09-29 20:53:35 on dist-test-slave-m9g1"
server_key: "c9135842b5c56460367d375373461902"
server_key_iv: "5608fcad0ba45178f64fe01b3ee37515"
server_key_version: "encryptionkey@0"
I20260929 20:53:35.195422 6316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:53:35.198514 6331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:35.199687 6316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 20:53:35.199837 6316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "101e97da2b5e49b8a64585e549180afd"
format_stamp: "Formatted at 2026-09-29 20:53:35 on dist-test-slave-m9g1"
server_key: "c9135842b5c56460367d375373461902"
server_key_iv: "5608fcad0ba45178f64fe01b3ee37515"
server_key_version: "encryptionkey@0"
I20260929 20:53:35.200001 6316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:35.219074 6316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:35.222637 6316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:35.222900 6316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:35.223752 6316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:35.224984 6316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:35.225055 6316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:35.225165 6316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:35.225215 6316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:35.253391 6316 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39519
I20260929 20:53:35.253436 6443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39519 every 8 connection(s)
I20260929 20:53:35.254611 6316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:53:35.260978 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 6316
I20260929 20:53:35.261148 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:53:35.261643 388 external_mini_cluster.cc:1428] Setting key e33972689fef4e4a1c571d79596c3328
Sep 29 20:53:35 dist-test-slave-m9g1 krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:35.272370 6179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:35.256895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39939 (local address 127.0.97.62:45313)
0929 20:53:35.257103 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:35.257109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:35.258215 (+ 1106us) server_negotiation.cc:407] Connection header received
0929 20:53:35.259874 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:35.259878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:35.259940 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:35.260022 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:35.261662 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:35.262437 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:35.264245 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:35.264646 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:35.264735 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:35.267806 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:35.267811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:35.267815 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:35.267845 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:35.270023 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:35.270652 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:35.270656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:35.270658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:35.270769 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:35.271101 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:35.271104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:35.271106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:35.271347 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:35.271451 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:35.271986 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 20:53:35.272139 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:53:35.273319 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:35.257387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45313 (local address 127.0.97.3:39939)
0929 20:53:35.258070 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:35.258106 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:35.259637 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:35.260204 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:35.260213 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:35.260737 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:35.261498 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:35.261524 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:35.262562 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:35.262566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:35.264114 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:35.264127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:35.264815 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:35.264819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:35.265003 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:35.265594 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:35.265619 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:35.267625 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:35.270148 (+ 2523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:35.270154 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:35.270166 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:35.270568 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:35.270855 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:35.270858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:35.270861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:35.271021 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:35.271433 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:35.271438 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:35.271709 (+ 271us) client_negotiation.cc:769] Sending connection context
0929 20:53:35.271914 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 20:53:35.272360 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":265,"threads_started":1}
I20260929 20:53:35.274577 6444 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45313
I20260929 20:53:35.274863 6444 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:35.275699 6444 heartbeater.cc:511] Master 127.0.97.62:45313 requested a full tablet report, sending...
I20260929 20:53:35.277070 5988 ts_manager.cc:194] Registered new tserver with Master: 101e97da2b5e49b8a64585e549180afd (127.0.97.3:39519)
I20260929 20:53:35.277810 5988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39939
I20260929 20:53:35.288151 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/test-admin.keytab.
Sep 29 20:53:35 dist-test-slave-m9g1 krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:35.310627 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 20:53:35.314939 6451 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:35.312911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36404 (local address 127.0.97.1:38769)
0929 20:53:35.313505 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:35.313512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:35.313540 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:53:35.313604 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:35.313611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:35.313811 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:35.313934 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:35.314627 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36404: BlockingRecv error: recv got EOF from 127.0.0.1:36404 (error 108)
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":361,"threads_started":1}
W20260929 20:53:35.315164 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/krb5cc' of type FILE
Sep 29 20:53:35 dist-test-slave-m9g1 krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:35.316725 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 20:53:35 dist-test-slave-m9g1 krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:35.317978 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 20:53:35 dist-test-slave-m9g1 krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:53:35.331681 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:35.319349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36418 (local address 127.0.97.1:38769)
0929 20:53:35.319482 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:35.319486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:35.319868 (+ 382us) server_negotiation.cc:407] Connection header received
0929 20:53:35.319983 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:35.319986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:35.320046 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:35.320118 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:35.320920 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:35.321866 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:35.323076 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:35.323619 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:35.323715 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:35.326967 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:35.326977 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:35.326990 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:35.327040 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:35.329526 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:35.330049 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:35.330055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:35.330057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:35.330172 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:35.330468 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:35.330471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:35.330473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:35.330788 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:35.331004 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:35.331207 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 20:53:35.331409 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 20:53:35.336787 6451 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:35.335421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36432 (local address 127.0.97.1:38769)
0929 20:53:35.335610 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:35.335616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:35.335735 (+ 119us) server_negotiation.cc:407] Connection header received
0929 20:53:35.335845 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:35.335848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:35.335908 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:35.335978 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:35.336590 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36432: BlockingRecv error: recv got EOF from 127.0.0.1:36432 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 20:53:35.336946 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/krb5cc' of type FILE
Sep 29 20:53:35 dist-test-slave-m9g1 krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:35.338311 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 20:53:35 dist-test-slave-m9g1 krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:35.339442 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 20:53:35 dist-test-slave-m9g1 krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:53:35.352339 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:35.340532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36434 (local address 127.0.97.1:38769)
0929 20:53:35.340683 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:35.340689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:35.340941 (+ 252us) server_negotiation.cc:407] Connection header received
0929 20:53:35.341079 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:35.341083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:35.341134 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:35.341206 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:35.342022 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:35.342754 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:35.343897 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:35.344302 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:35.344403 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:35.347568 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:35.347575 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:35.347580 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:35.347616 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:35.350145 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:35.350840 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:35.350843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:35.350846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:35.350976 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:35.351279 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:35.351282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:35.351284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:35.351535 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:35.351708 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:35.351940 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 20:53:35.352083 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 20:53:35.356227 6451 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:35.354881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36444 (local address 127.0.97.1:38769)
0929 20:53:35.355144 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:35.355150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:35.355196 (+ 46us) server_negotiation.cc:407] Connection header received
0929 20:53:35.355319 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:35.355323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:35.355405 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:35.355492 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:35.356013 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36444: BlockingRecv error: recv got EOF from 127.0.0.1:36444 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
W20260929 20:53:35.356525 388 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715169435170-388-0/krb5kdc/krb5cc)
Sep 29 20:53:35 dist-test-slave-m9g1 krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:35.357969 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 20:53:35 dist-test-slave-m9g1 krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:35.359128 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 20:53:35 dist-test-slave-m9g1 krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:53:35.371915 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:35.360338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36452 (local address 127.0.97.1:38769)
0929 20:53:35.360505 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:35.360510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:35.360736 (+ 226us) server_negotiation.cc:407] Connection header received
0929 20:53:35.360878 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:35.360881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:35.360932 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:35.361005 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:35.361906 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:35.362641 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:35.363822 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:35.364212 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:35.364312 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:35.367356 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:35.367360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:35.367363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:35.367392 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:35.369605 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:35.370151 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:35.370157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:35.370161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:35.370335 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:35.370654 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:35.370660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:35.370663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:35.370928 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:35.371193 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:35.371447 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 20:53:35.371654 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 20:53:35.373271 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 6049
I20260929 20:53:35.378973 6177 heartbeater.cc:503] Master 127.0.97.62:45313 was elected leader, sending a full tablet report...
I20260929 20:53:35.383455 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:35.373886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36460 (local address 127.0.97.1:38769)
0929 20:53:35.374030 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:35.374036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:35.374343 (+ 307us) server_negotiation.cc:407] Connection header received
0929 20:53:35.374455 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:35.374459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:35.374507 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:35.374574 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:35.375304 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:35.375920 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:35.376973 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:35.377354 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:35.377438 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:35.379412 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:35.379416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:35.379419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:35.379444 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:35.381515 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:35.382026 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:35.382029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:35.382031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:35.382145 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:35.382447 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:35.382450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:35.382452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:35.382672 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:35.382844 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:35.383037 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 20:53:35.383177 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 20:53:35.383908 6172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:35.464252 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 6049
I20260929 20:53:35.477090 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 6183
Sep 29 20:53:35 dist-test-slave-m9g1 krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:53:35.490993 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:35.477956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46548 (local address 127.0.97.2:38347)
0929 20:53:35.478472 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:35.478479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:35.478503 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:53:35.478559 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:35.478565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:35.478766 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:35.478903 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:35.479829 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:35.480799 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:35.481945 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:35.482544 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:35.482654 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:35.486021 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:35.486031 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:35.486044 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:35.486073 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:35.488483 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:35.489070 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:35.489078 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:35.489080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:35.489189 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:35.489521 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:35.489526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:35.489531 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:35.489842 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:35.490117 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:35.490458 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 20:53:35.490686 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":277,"threads_started":1}
I20260929 20:53:35.492166 6306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:35.571254 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 6183
I20260929 20:53:35.583207 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 6316
Sep 29 20:53:35 dist-test-slave-m9g1 krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:53:35.609081 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:35.583969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.0.97.3:39519)
0929 20:53:35.584524 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:35.584531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:35.584558 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:53:35.584615 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:35.584621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:35.584826 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:35.584954 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:35.585801 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:35.586861 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:35.588092 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:35.588643 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:35.588738 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:35.604384 (+ 15646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:35.604398 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:35.604414 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:35.604448 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:35.606895 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:35.607471 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:35.607478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:35.607480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:35.607592 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:35.607850 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:35.607853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:35.607855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:35.608175 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:35.608396 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:35.608568 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 20:53:35.608774 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":258,"threads_started":1}
I20260929 20:53:35.610265 6439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:35.689162 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 6316
I20260929 20:53:35.701483 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 5956
I20260929 20:53:35.702601 6018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:35.801052 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 5956
2026-09-29T20:53:35Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3579 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:53:35 dist-test-slave-m9g1 krb5kdc[6471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:53:35 dist-test-slave-m9g1 krb5kdc[6471](info): set up 2 sockets
Sep 29 20:53:35 dist-test-slave-m9g1 krb5kdc[6471](info): commencing operation
krb5kdc: starting...
W20260929 20:53:36.786576 388 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:53:36 dist-test-slave-m9g1 krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:53:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:53:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260929 20:53:36.965698 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34335
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:37.262877 6484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:37.263134 6484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:37.263268 6484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:37.270090 6484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:53:37.270201 6484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:37.270285 6484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:37.270360 6484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:53:37.270408 6484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:53:37.279657 6484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:37.280860 6484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:37.282150 6484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:37.292579 6492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:37.292608 6490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:37.292693 6489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:37.293062 6484 server_base.cc:1034] running on GCE node
I20260929 20:53:37.293857 6484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:37.295181 6484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:37.296392 6484 hybrid_clock.cc:648] HybridClock initialized: now 1790715217296331 us; error 50 us; skew 500 ppm
I20260929 20:53:37.296697 6484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:37 dist-test-slave-m9g1 krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715217, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:37.299741 6484 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260929 20:53:37.301021 6484 webserver.cc:460] Webserver started at http://127.0.97.62:34023/ using document root <none> and password file <none>
I20260929 20:53:37.301615 6484 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:37.301710 6484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:37.301949 6484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:37.304653 6484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "347ec01fc4094722a63b6055158d723c"
format_stamp: "Formatted at 2026-09-29 20:53:37 on dist-test-slave-m9g1"
server_key: "42ea6d00ca24277ade857275d8c9e838"
server_key_iv: "5a665ed482c92c7d8a4319a481778e21"
server_key_version: "encryptionkey@0"
I20260929 20:53:37.305330 6484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "347ec01fc4094722a63b6055158d723c"
format_stamp: "Formatted at 2026-09-29 20:53:37 on dist-test-slave-m9g1"
server_key: "42ea6d00ca24277ade857275d8c9e838"
server_key_iv: "5a665ed482c92c7d8a4319a481778e21"
server_key_version: "encryptionkey@0"
I20260929 20:53:37.309957 6484 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 20:53:37.313035 6499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:37.314245 6484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 20:53:37.314376 6484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "347ec01fc4094722a63b6055158d723c"
format_stamp: "Formatted at 2026-09-29 20:53:37 on dist-test-slave-m9g1"
server_key: "42ea6d00ca24277ade857275d8c9e838"
server_key_iv: "5a665ed482c92c7d8a4319a481778e21"
server_key_version: "encryptionkey@0"
I20260929 20:53:37.314519 6484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:37.336436 6484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:37.340296 6484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:37.340521 6484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:37.356799 6484 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34335
I20260929 20:53:37.356838 6550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34335 every 8 connection(s)
I20260929 20:53:37.358144 6484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:53:37.360535 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 6484
I20260929 20:53:37.360731 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:53:37.361340 388 external_mini_cluster.cc:1428] Setting key 68c0472ae00e0d50f4af585ff2e3c212
I20260929 20:53:37.362058 6551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:53:37 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:53:37.375173 6551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c: Bootstrap starting.
I20260929 20:53:37.378218 6551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:37.379402 6551 log.cc:824] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:37.381958 6551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c: No bootstrap required, opened a new log
I20260929 20:53:37.382266 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:37.363231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51840 (local address 127.0.97.62:34335)
0929 20:53:37.363993 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:37.364009 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:37.364056 (+ 47us) server_negotiation.cc:407] Connection header received
0929 20:53:37.365574 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:37.365601 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:37.366108 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:37.366559 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:37.367680 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:37.369623 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:37.370785 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:37.371816 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:37.371997 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:37.375714 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:37.375721 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:37.375737 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:37.375773 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:37.378542 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:37.379301 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:37.379320 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:37.379326 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:37.379477 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:37.379834 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:37.379837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:37.379839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:37.380320 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:37.380575 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:37.380975 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 20:53:37.381334 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":296,"threads_started":1}
I20260929 20:53:37.389421 6551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } }
I20260929 20:53:37.389807 6551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:37.389896 6551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 347ec01fc4094722a63b6055158d723c, State: Initialized, Role: FOLLOWER
I20260929 20:53:37.390466 6551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } }
I20260929 20:53:37.390676 6551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:53:37.390795 6551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:53:37.390939 6551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:37.392851 6551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } }
I20260929 20:53:37.393291 6551 leader_election.cc:302] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 347ec01fc4094722a63b6055158d723c; no voters:
I20260929 20:53:37.393868 6551 leader_election.cc:288] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:53:37.393997 6556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:37.394681 6556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [term 1 LEADER]: Becoming Leader. State: Replica: 347ec01fc4094722a63b6055158d723c, State: Running, Role: LEADER
I20260929 20:53:37.395219 6556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } }
I20260929 20:53:37.396051 6551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:53:37.397826 6557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "347ec01fc4094722a63b6055158d723c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } }
I20260929 20:53:37.398208 6557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:37.399418 6558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 347ec01fc4094722a63b6055158d723c. Latest consensus state: current_term: 1 leader_uuid: "347ec01fc4094722a63b6055158d723c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } }
I20260929 20:53:37.399760 6558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:37.400771 6565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:53:37.405460 6565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:53:37.413789 6565 catalog_manager.cc:1347] Generated new cluster ID: 4418e57d94d541228f2db15ec1ae6260
I20260929 20:53:37.413877 6565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:53:37.449120 6565 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:53:37.450964 6565 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:53:37.464023 6565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c: Generated new TSK 0
I20260929 20:53:37.465040 6565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260929 20:53:37.518656 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34335
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:37.825348 6577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:37.825598 6577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:37.825719 6577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:37.832686 6577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:37.832791 6577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:37.832939 6577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:53:37.842468 6577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:37.843806 6577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:37.845052 6577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:37.857007 6583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:37.857040 6585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:37.857122 6582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:37.857976 6577 server_base.cc:1034] running on GCE node
I20260929 20:53:37.858529 6577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:37.859380 6577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:37.860558 6577 hybrid_clock.cc:648] HybridClock initialized: now 1790715217860507 us; error 44 us; skew 500 ppm
I20260929 20:53:37.860834 6577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:37 dist-test-slave-m9g1 krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715217, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:37.863686 6577 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 20:53:37.864880 6577 webserver.cc:460] Webserver started at http://127.0.97.1:44769/ using document root <none> and password file <none>
I20260929 20:53:37.865444 6577 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:37.865532 6577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:37.865756 6577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:37.868402 6577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "e816c455c3944cac8973eaac1bf8580a"
format_stamp: "Formatted at 2026-09-29 20:53:37 on dist-test-slave-m9g1"
server_key: "e090b4df7ca38c0a7e02fbab26e545fa"
server_key_iv: "92523ead42251627220dab698208f63b"
server_key_version: "encryptionkey@0"
I20260929 20:53:37.869093 6577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e816c455c3944cac8973eaac1bf8580a"
format_stamp: "Formatted at 2026-09-29 20:53:37 on dist-test-slave-m9g1"
server_key: "e090b4df7ca38c0a7e02fbab26e545fa"
server_key_iv: "92523ead42251627220dab698208f63b"
server_key_version: "encryptionkey@0"
I20260929 20:53:37.873651 6577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:53:37.876802 6592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:37.878008 6577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:53:37.878139 6577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "e816c455c3944cac8973eaac1bf8580a"
format_stamp: "Formatted at 2026-09-29 20:53:37 on dist-test-slave-m9g1"
server_key: "e090b4df7ca38c0a7e02fbab26e545fa"
server_key_iv: "92523ead42251627220dab698208f63b"
server_key_version: "encryptionkey@0"
I20260929 20:53:37.878336 6577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:37.916144 6577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:37.919991 6577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:37.920218 6577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:37.921028 6577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:37.922242 6577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:37.922309 6577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:37.922399 6577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:37.922439 6577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:37.949915 6577 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46367
I20260929 20:53:37.949963 6704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46367 every 8 connection(s)
I20260929 20:53:37.951122 6577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:53:37.957316 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 6577
I20260929 20:53:37.957484 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:53:37.958112 388 external_mini_cluster.cc:1428] Setting key caba9ef55689a6205428d1810ccf6fd0
Sep 29 20:53:37 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715217, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:53:37.969703 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:37.953444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57165 (local address 127.0.97.62:34335)
0929 20:53:37.953879 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:37.953886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:37.954817 (+ 931us) server_negotiation.cc:407] Connection header received
0929 20:53:37.956458 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:37.956462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:37.956522 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:37.956612 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:37.958002 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:37.958971 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:37.960740 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:37.961306 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:37.961426 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:37.964490 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:37.964496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:37.964500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:37.964535 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:37.967027 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:37.967773 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:37.967777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:37.967779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:37.967879 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:37.968266 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:37.968269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:37.968271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:37.968500 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:37.968599 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:37.969308 (+ 709us) server_negotiation.cc:299] Negotiation successful
0929 20:53:37.969464 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":209,"threads_started":1}
I20260929 20:53:37.971056 6708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:37.954006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.1:57165)
0929 20:53:37.954679 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:37.954714 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:37.956227 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:37.956792 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:37.956799 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:37.957277 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:37.957847 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:37.957865 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:37.959114 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:37.959120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:37.960608 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:37.960620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:37.961479 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:37.961484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:37.961679 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:37.962285 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:37.962310 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:37.964295 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:37.967160 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:37.967169 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:37.967185 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:37.967682 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:37.967963 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:37.967968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:37.967971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:37.968181 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:37.968600 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:37.968608 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:37.968968 (+ 360us) client_negotiation.cc:769] Sending connection context
0929 20:53:37.969262 (+ 294us) client_negotiation.cc:240] Negotiation successful
0929 20:53:37.969716 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":278,"threads_started":1}
I20260929 20:53:37.972496 6705 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34335
I20260929 20:53:37.972764 6705 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:37.973469 6705 heartbeater.cc:511] Master 127.0.97.62:34335 requested a full tablet report, sending...
I20260929 20:53:37.975584 6516 ts_manager.cc:194] Registered new tserver with Master: e816c455c3944cac8973eaac1bf8580a (127.0.97.1:46367)
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:53:37.977526 6516 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260929 20:53:38.001504 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34335
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:38.302379 6711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:38.302644 6711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:38.302776 6711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:38.309746 6711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:38.309849 6711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:38.309995 6711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:53:38.319592 6711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:38.320878 6711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:38.322172 6711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:38.334054 6716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:38.334126 6719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:38.334153 6717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:38.335183 6711 server_base.cc:1034] running on GCE node
I20260929 20:53:38.335731 6711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:38.336587 6711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:38.337777 6711 hybrid_clock.cc:648] HybridClock initialized: now 1790715218337722 us; error 46 us; skew 500 ppm
I20260929 20:53:38.338083 6711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:38 dist-test-slave-m9g1 krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715218, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:38.343892 6711 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 20:53:38.345450 6711 webserver.cc:460] Webserver started at http://127.0.97.2:37373/ using document root <none> and password file <none>
I20260929 20:53:38.346091 6711 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:38.346215 6711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:38.346494 6711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:38.349300 6711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "9f3bebe27bd744c1bcd36918f642ca91"
format_stamp: "Formatted at 2026-09-29 20:53:38 on dist-test-slave-m9g1"
server_key: "b6c1915516a47486f8b9fd1bc0d318ac"
server_key_iv: "c1959e1310f0031b8fbc5e65abeaf223"
server_key_version: "encryptionkey@0"
I20260929 20:53:38.350006 6711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9f3bebe27bd744c1bcd36918f642ca91"
format_stamp: "Formatted at 2026-09-29 20:53:38 on dist-test-slave-m9g1"
server_key: "b6c1915516a47486f8b9fd1bc0d318ac"
server_key_iv: "c1959e1310f0031b8fbc5e65abeaf223"
server_key_version: "encryptionkey@0"
I20260929 20:53:38.355053 6711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:53:38.358392 6726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:38.359712 6711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:53:38.359845 6711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "9f3bebe27bd744c1bcd36918f642ca91"
format_stamp: "Formatted at 2026-09-29 20:53:38 on dist-test-slave-m9g1"
server_key: "b6c1915516a47486f8b9fd1bc0d318ac"
server_key_iv: "c1959e1310f0031b8fbc5e65abeaf223"
server_key_version: "encryptionkey@0"
I20260929 20:53:38.360005 6711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:38.378600 6711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:38.382309 6711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:38.382560 6711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:38.383440 6711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:38.384663 6711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:38.384730 6711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:38.384843 6711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:38.384889 6711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:38.411607 6711 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36383
I20260929 20:53:38.411646 6838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36383 every 8 connection(s)
I20260929 20:53:38.412786 6711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:53:38.417822 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 6711
I20260929 20:53:38.418006 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:53:38.418568 388 external_mini_cluster.cc:1428] Setting key 9cebbb7f3c8e5eacd293d731eaf93286
Sep 29 20:53:38 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715218, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:53:38.432175 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:38.415181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45825 (local address 127.0.97.62:34335)
0929 20:53:38.415383 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:38.415390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:38.416473 (+ 1083us) server_negotiation.cc:407] Connection header received
0929 20:53:38.418544 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:38.418549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:38.418615 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:38.418718 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:38.420348 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:38.421059 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:38.422614 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:38.423011 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:38.423117 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:38.426526 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:38.426532 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:38.426545 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:38.426583 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:38.429473 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:38.430110 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:38.430115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:38.430118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:38.430284 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:38.430634 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:38.430638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:38.430642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:38.430941 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:38.431068 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:38.431665 (+ 597us) server_negotiation.cc:299] Negotiation successful
0929 20:53:38.431865 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:53:38.433003 6841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:38.415662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.2:45825)
0929 20:53:38.416318 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:38.416359 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:38.418230 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:38.418956 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:38.418964 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:38.419589 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:38.420202 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:38.420222 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:38.421201 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:38.421205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:38.422485 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:38.422496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:38.423218 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:38.423225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:38.423433 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:38.424054 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:38.424079 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:38.426310 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:38.429600 (+ 3290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:38.429606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:38.429619 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:38.430006 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:38.430370 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:38.430373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:38.430375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:38.430535 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:38.431035 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:38.431040 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:38.431332 (+ 292us) client_negotiation.cc:769] Sending connection context
0929 20:53:38.431542 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 20:53:38.431904 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":246,"threads_started":1}
I20260929 20:53:38.434417 6839 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34335
I20260929 20:53:38.434686 6839 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:38.435328 6839 heartbeater.cc:511] Master 127.0.97.62:34335 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:53:38.436731 6516 ts_manager.cc:194] Registered new tserver with Master: 9f3bebe27bd744c1bcd36918f642ca91 (127.0.97.2:36383)
I20260929 20:53:38.437486 6516 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 20:53:38.459125 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34335
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:38.769661 6844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:38.769944 6844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:38.770072 6844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:38.777477 6844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:38.777603 6844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:38.777765 6844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:53:38.787359 6844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:38.788568 6844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:38.789813 6844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:38.801963 6849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:38.801930 6850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:38.802000 6852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:38.802968 6844 server_base.cc:1034] running on GCE node
I20260929 20:53:38.803519 6844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:38.804382 6844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:38.805577 6844 hybrid_clock.cc:648] HybridClock initialized: now 1790715218805527 us; error 44 us; skew 500 ppm
I20260929 20:53:38.805860 6844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:38 dist-test-slave-m9g1 krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715218, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:38.808648 6844 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 20:53:38.809930 6844 webserver.cc:460] Webserver started at http://127.0.97.3:45527/ using document root <none> and password file <none>
I20260929 20:53:38.810585 6844 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:38.810693 6844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:38.810951 6844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:38.813648 6844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "3529304287894649b2750ee4aeb0fa05"
format_stamp: "Formatted at 2026-09-29 20:53:38 on dist-test-slave-m9g1"
server_key: "3aa9411c01ef247d029b0fcb1ab1a128"
server_key_iv: "7d0e3614b84cab17cfa9eee7cd4b1046"
server_key_version: "encryptionkey@0"
I20260929 20:53:38.814404 6844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3529304287894649b2750ee4aeb0fa05"
format_stamp: "Formatted at 2026-09-29 20:53:38 on dist-test-slave-m9g1"
server_key: "3aa9411c01ef247d029b0fcb1ab1a128"
server_key_iv: "7d0e3614b84cab17cfa9eee7cd4b1046"
server_key_version: "encryptionkey@0"
I20260929 20:53:38.819168 6844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:53:38.822220 6859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:38.823472 6844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 20:53:38.823609 6844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "3529304287894649b2750ee4aeb0fa05"
format_stamp: "Formatted at 2026-09-29 20:53:38 on dist-test-slave-m9g1"
server_key: "3aa9411c01ef247d029b0fcb1ab1a128"
server_key_iv: "7d0e3614b84cab17cfa9eee7cd4b1046"
server_key_version: "encryptionkey@0"
I20260929 20:53:38.823771 6844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:38.849222 6844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:38.852918 6844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:38.853185 6844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:38.854092 6844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:38.855475 6844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:38.855548 6844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:38.855662 6844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:38.855713 6844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:38.883701 6844 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45645
I20260929 20:53:38.883746 6971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45645 every 8 connection(s)
I20260929 20:53:38.884950 6844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:53:38.895143 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 6844
I20260929 20:53:38.895290 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:53:38.895777 388 external_mini_cluster.cc:1428] Setting key 10836b362bc50e5728b125e1309b8b02
Sep 29 20:53:38 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715218, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:53:38.904120 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:38.887404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49347 (local address 127.0.97.62:34335)
0929 20:53:38.887580 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:38.887584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:38.888695 (+ 1111us) server_negotiation.cc:407] Connection header received
0929 20:53:38.890450 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:38.890455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:38.890538 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:38.890624 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:38.892047 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:38.892750 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:38.894221 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:38.894625 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:38.894729 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:38.898985 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:38.898991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:38.898996 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:38.899034 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:38.901651 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:38.902332 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:38.902336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:38.902338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:38.902447 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:38.902767 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:38.902770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:38.902772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:38.903045 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:38.903146 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:38.903737 (+ 591us) server_negotiation.cc:299] Negotiation successful
0929 20:53:38.903889 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:53:38.905051 6974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:38.887881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.3:49347)
0929 20:53:38.888550 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:38.888587 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:38.890152 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:38.890859 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:38.890867 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:38.891354 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:38.891888 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:38.891904 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:38.892891 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:38.892895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:38.894064 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:38.894079 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:38.894822 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:38.894830 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:38.895054 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:38.895754 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:38.895782 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:38.898766 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:38.901785 (+ 3019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:38.901792 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:38.901805 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:38.902246 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:38.902523 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:38.902526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:38.902528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:38.902689 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:38.903133 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:38.903138 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:38.903437 (+ 299us) client_negotiation.cc:769] Sending connection context
0929 20:53:38.903659 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 20:53:38.904033 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":246,"threads_started":1}
I20260929 20:53:38.906222 6972 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34335
I20260929 20:53:38.906507 6972 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:38.907145 6972 heartbeater.cc:511] Master 127.0.97.62:34335 requested a full tablet report, sending...
I20260929 20:53:38.908456 6516 ts_manager.cc:194] Registered new tserver with Master: 3529304287894649b2750ee4aeb0fa05 (127.0.97.3:45645)
I20260929 20:53:38.909250 6516 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49347
I20260929 20:53:38.911381 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260929 20:53:38.953702 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40257
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34335,127.0.97.61:40257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:53:38.980417 6705 heartbeater.cc:503] Master 127.0.97.62:34335 was elected leader, sending a full tablet report...
W20260929 20:53:39.255191 6977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:39.255467 6977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:39.255596 6977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:39.262578 6977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:53:39.262669 6977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:39.262737 6977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:39.262804 6977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:53:39.262861 6977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:53:39.272635 6977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34335,127.0.97.61:40257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
W20260929 20:53:39.272976 6977 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:34335,127.0.97.61:40257'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 20:53:39.273962 6977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:39.275295 6977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:39.285827 6985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:39.285959 6982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:39.285907 6983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:39.286703 6977 server_base.cc:1034] running on GCE node
I20260929 20:53:39.287389 6977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:39.288633 6977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:39.289842 6977 hybrid_clock.cc:648] HybridClock initialized: now 1790715219289787 us; error 29 us; skew 500 ppm
I20260929 20:53:39.290113 6977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:39 dist-test-slave-m9g1 krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715219, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:39.292990 6977 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260929 20:53:39.294250 6977 webserver.cc:460] Webserver started at http://127.0.97.61:37573/ using document root <none> and password file <none>
I20260929 20:53:39.294863 6977 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:39.294941 6977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:39.295177 6977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:39.298043 6977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/data/instance:
uuid: "60d848cea40b4d6e9c79fdda2fa9c401"
format_stamp: "Formatted at 2026-09-29 20:53:39 on dist-test-slave-m9g1"
server_key: "678639fb6f6c55ea404e5765377217ae"
server_key_iv: "82fd7e3b3bc14240a68a4f55f5b793cb"
server_key_version: "encryptionkey@0"
I20260929 20:53:39.298776 6977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/wal/instance:
uuid: "60d848cea40b4d6e9c79fdda2fa9c401"
format_stamp: "Formatted at 2026-09-29 20:53:39 on dist-test-slave-m9g1"
server_key: "678639fb6f6c55ea404e5765377217ae"
server_key_iv: "82fd7e3b3bc14240a68a4f55f5b793cb"
server_key_version: "encryptionkey@0"
I20260929 20:53:39.303558 6977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 20:53:39.306648 6992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:39.307875 6977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 20:53:39.307998 6977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/wal
uuid: "60d848cea40b4d6e9c79fdda2fa9c401"
format_stamp: "Formatted at 2026-09-29 20:53:39 on dist-test-slave-m9g1"
server_key: "678639fb6f6c55ea404e5765377217ae"
server_key_iv: "82fd7e3b3bc14240a68a4f55f5b793cb"
server_key_version: "encryptionkey@0"
I20260929 20:53:39.308162 6977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:39.318583 6977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:39.322294 6977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:39.322500 6977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:39.338536 6977 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40257
I20260929 20:53:39.338555 7043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40257 every 8 connection(s)
I20260929 20:53:39.339805 6977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/data/info.pb
I20260929 20:53:39.340651 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 6977
I20260929 20:53:39.343876 7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:39.350417 7044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:53:39 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 20:53:39 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715219, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:53:39.363125 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:39.342751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34948 (local address 127.0.97.61:40257)
0929 20:53:39.343451 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:39.343463 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:39.343502 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:53:39.345114 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:39.345137 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:39.345576 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:39.345931 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:39.347342 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.348660 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.349848 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.350550 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.350732 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:39.355108 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:39.355115 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:39.355124 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:39.355159 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:39.358694 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:39.359524 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:39.359532 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:39.359541 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:39.359731 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:39.360166 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:39.360171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:39.360175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:39.360639 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:39.360900 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:39.361517 (+ 617us) server_negotiation.cc:299] Negotiation successful
0929 20:53:39.361933 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":307,"threads_started":1}
I20260929 20:53:39.367798 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:39.352526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.0.1:51842)
0929 20:53:39.353157 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:39.353211 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:39.353373 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:39.353725 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:39.353729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:39.354441 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:39.354755 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:39.354769 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.355772 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.355778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:39.357138 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:39.357150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.357917 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.357924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:39.358097 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:39.358847 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:39.358876 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:39.361421 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:39.365117 (+ 3696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:39.365126 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:39.365133 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:39.365646 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:39.366145 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:39.366148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:39.366151 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:39.366315 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:39.367055 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:39.367061 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:39.367205 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 20:53:39.367343 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 20:53:39.367541 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":272,"threads_started":1}
I20260929 20:53:39.367959 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:39.352671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51842 (local address 127.0.97.62:34335)
0929 20:53:39.352865 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:39.352871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:39.353312 (+ 441us) server_negotiation.cc:407] Connection header received
0929 20:53:39.353487 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:39.353492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:39.353563 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:39.353671 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:39.354900 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.355650 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.357273 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.357739 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.357843 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:39.361611 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:39.361617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:39.361621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:39.361663 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:39.364992 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:39.365748 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:39.365754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:39.365758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:39.365918 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:39.366548 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:39.366553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:39.366556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:39.366947 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:39.367079 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:39.367473 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 20:53:39.367669 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:53:39.369401 7044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:53:39 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715219, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 20:53:39.379729 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:39.370071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40257 (local address 127.0.0.1:34954)
0929 20:53:39.370257 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:39.370273 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:39.370344 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:39.370599 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:39.370603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:39.370906 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:39.371024 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:39.371031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.371895 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.371898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:39.372718 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:39.372727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.373285 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.373289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:39.373425 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:39.373889 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:39.373904 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:39.375634 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:39.377913 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:39.377917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:39.377919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:39.378267 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:39.378560 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:39.378563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:39.378565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:39.378703 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:39.379103 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:39.379106 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:39.379211 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 20:53:39.379315 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 20:53:39.379490 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 20:53:39.379878 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:39.370121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.0.97.61:40257)
0929 20:53:39.370287 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:39.370291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:39.370308 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:53:39.370418 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:39.370421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:39.370468 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:39.370537 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:39.371137 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.371803 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.372818 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.373182 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.373268 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:39.375798 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:39.375803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:39.375806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:39.375831 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:39.377812 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:39.378350 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:39.378354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:39.378356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:39.378477 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:39.378786 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:39.378790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:39.378792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:39.379015 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:39.379110 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:39.379396 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 20:53:39.379600 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":46}
I20260929 20:53:39.384178 7044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: Bootstrap starting.
I20260929 20:53:39.387044 7044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:39.388361 7044 log.cc:824] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:39.390749 7044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: No bootstrap required, opened a new log
I20260929 20:53:39.397254 7044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } }
I20260929 20:53:39.397579 7044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:39.397660 7044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d848cea40b4d6e9c79fdda2fa9c401, State: Initialized, Role: FOLLOWER
I20260929 20:53:39.398161 7044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } }
I20260929 20:53:39.398973 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } } }
I20260929 20:53:39.399240 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:53:39.399885 7044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 20:53:39.407758 7061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:53:39.407871 7061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:53:39.410068 6977 master_runner.cc:428] Detected that this master 60d848cea40b4d6e9c79fdda2fa9c401 is joining an existing cluster
I20260929 20:53:39.410166 6977 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:40257
I20260929 20:53:39.440052 6839 heartbeater.cc:503] Master 127.0.97.62:34335 was elected leader, sending a full tablet report...
I20260929 20:53:39.441324 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:39.414727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51854 (local address 127.0.97.62:34335)
0929 20:53:39.414908 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:39.414914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:39.426756 (+ 11842us) server_negotiation.cc:407] Connection header received
0929 20:53:39.426916 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:39.426921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:39.426998 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:39.427085 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:39.428547 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.429671 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.431826 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.432437 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.432531 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:39.434970 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:39.434977 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:39.434982 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:39.435020 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:39.438280 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:39.439106 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:39.439111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:39.439115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:39.439266 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:39.439721 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:39.439725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:39.439729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:39.440066 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:39.440176 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:39.440796 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 20:53:39.440982 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:53:39.442233 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:39.414326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.0.1:51854)
0929 20:53:39.426641 (+ 12315us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:39.426660 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:39.426807 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:39.427267 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:39.427272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:39.427983 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:39.428377 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:39.428391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.429843 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.429849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:39.431662 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:39.431676 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.432802 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.432808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:39.432973 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:39.433615 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:39.433638 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:39.434766 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:39.438417 (+ 3651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:39.438421 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:39.438425 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:39.438975 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:39.439410 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:39.439415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:39.439421 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:39.439583 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:39.440249 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:39.440253 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:39.440459 (+ 206us) client_negotiation.cc:769] Sending connection context
0929 20:53:39.441527 (+ 1068us) client_negotiation.cc:240] Negotiation successful
0929 20:53:39.441836 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12171,"thread_start_us":14785,"threads_started":1}
I20260929 20:53:39.442737 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:39.414357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34966 (local address 127.0.97.61:40257)
0929 20:53:39.414548 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:39.414554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:39.429495 (+ 14941us) server_negotiation.cc:407] Connection header received
0929 20:53:39.429652 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:39.429657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:39.429714 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:39.429764 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:39.431094 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.432051 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.433730 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.434364 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.434444 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:39.436599 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:39.436605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:39.436609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:39.436655 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:39.439989 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:39.440733 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:39.440738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:39.440741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:39.440888 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:39.441278 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:39.441283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:39.441286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:39.441598 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:39.441735 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:39.442239 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 20:53:39.442429 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:53:39.443616 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:39.414810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40257 (local address 127.0.0.1:34966)
0929 20:53:39.429406 (+ 14596us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:39.429420 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:39.429528 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:39.429909 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:39.429913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:39.430666 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:39.430966 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:39.430976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.432228 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.432233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:39.433594 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:39.433607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.434728 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.434734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:39.434883 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:39.435419 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:39.435441 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:39.436427 (+ 986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:39.440118 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:39.440122 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:39.440125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:39.440612 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:39.441023 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:39.441027 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:39.441030 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:39.441165 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:39.441814 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:39.441819 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:39.441956 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 20:53:39.442945 (+ 989us) client_negotiation.cc:240] Negotiation successful
0929 20:53:39.443221 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14464,"thread_start_us":11534,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 29 20:53:39 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715217, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 20:53:39.465807 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:39.447451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34982 (local address 127.0.97.61:40257)
0929 20:53:39.447627 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:39.447632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:39.450809 (+ 3177us) server_negotiation.cc:407] Connection header received
0929 20:53:39.451166 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:39.451171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:39.451254 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:39.451351 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:39.452604 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.453535 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.455421 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.455959 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.456066 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:39.459370 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:39.459374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:39.459378 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:39.459405 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:39.462519 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:39.463441 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:39.463446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:39.463449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:39.463599 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:39.464097 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:39.464101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:39.464104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:39.464490 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:39.464624 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:39.465220 (+ 596us) server_negotiation.cc:299] Negotiation successful
0929 20:53:39.465447 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:53:39.465822 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:39.447451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40257 (local address 127.0.0.1:34982)
0929 20:53:39.450677 (+ 3226us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:39.450726 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:39.450945 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:39.451419 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:39.451427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:39.452197 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:39.452457 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:39.452470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.453665 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.453671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:39.455267 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:39.455282 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.456111 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.456117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:39.456303 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:39.456870 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:39.456890 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:39.459215 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:39.462654 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:39.462667 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:39.462675 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:39.463307 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:39.463710 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:39.463716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:39.463720 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:39.463911 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:39.464694 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:39.464704 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:39.464889 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 20:53:39.465075 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 20:53:39.465403 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2993,"thread_start_us":2861,"threads_started":1}
I20260929 20:53:39.468323 6516 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:40257: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 40257
}
attrs {
promote: true
}
}
dest_uuid: "347ec01fc4094722a63b6055158d723c"
cas_config_opid_index: -1
I20260929 20:53:39.469444 6516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: true } }
I20260929 20:53:39.473032 7073 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 60d848cea40b4d6e9c79fdda2fa9c401 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: true } } }
I20260929 20:53:39.473901 7019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:39.477366 7073 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:40257
I20260929 20:53:39.477487 7019 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 1 FOLLOWER]: Refusing update from remote peer 347ec01fc4094722a63b6055158d723c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 20:53:39.478154 6977 master.cc:537] Master@127.0.97.61:40257 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260929 20:53:39.482280 7074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:39.482967 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41713
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34335,127.0.97.61:40257,127.0.97.60:41713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:53:39.490747 6977 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:39.491212 6977 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:39.491463 6977 tablet_replica.cc:330] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: stopping tablet replica
W20260929 20:53:39.496099 6503 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 20:53:39.497138 7075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "347ec01fc4094722a63b6055158d723c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: true } } }
I20260929 20:53:39.497474 7075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:39.503841 6977 master.cc:552] Master@127.0.97.61:40257 shutdown complete.
I20260929 20:53:39.504607 6977 master_runner.cc:305] Clearing existing system tablet
I20260929 20:53:39.505160 6977 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:39.505787 6977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:39.511716 6977 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.001s
I20260929 20:53:39.511897 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:39.511449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40257 (local address 127.0.0.1:34992)
0929 20:53:39.511630 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:39.511736 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 20:53:39.512401 6503 consensus_peers.cc:487] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c -> Peer 60d848cea40b4d6e9c79fdda2fa9c401 (127.0.97.61:40257): Couldn't send request to peer 60d848cea40b4d6e9c79fdda2fa9c401. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 20:53:39.513316 7081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:39.513700 6977 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:39.513953 6977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 20:53:39.514034 6977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/wal
uuid: "60d848cea40b4d6e9c79fdda2fa9c401"
format_stamp: "Formatted at 2026-09-29 20:53:39 on dist-test-slave-m9g1"
server_key: "678639fb6f6c55ea404e5765377217ae"
server_key_iv: "82fd7e3b3bc14240a68a4f55f5b793cb"
server_key_version: "encryptionkey@0"
I20260929 20:53:39.518965 6977 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:39.525544 6977 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 20:53:39.525708 6977 log.cc:1196] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 20:53:39.526110 6977 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: Deleting consensus metadata
I20260929 20:53:39.526381 6977 master_runner.cc:315] Copying system tablet from 127.0.97.62:34335
I20260929 20:53:39.528690 6977 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:34335
I20260929 20:53:39.543403 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:39.529396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.0.1:51864)
0929 20:53:39.529854 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:39.529876 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:39.530027 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:39.530431 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:39.530436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:39.530947 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:39.531214 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:39.531227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.532634 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.532640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:39.534134 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:39.534149 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.535041 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.535047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:39.535236 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:39.535886 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:39.535909 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:39.536946 (+ 1037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:39.540555 (+ 3609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:39.540561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:39.540565 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:39.541056 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:39.541489 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:39.541495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:39.541498 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:39.541673 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:39.542328 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:39.542334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:39.542504 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 20:53:39.542652 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 20:53:39.542949 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":256,"threads_started":1}
I20260929 20:53:39.543481 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:39.529456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51864 (local address 127.0.97.62:34335)
0929 20:53:39.529655 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:39.529661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:39.529976 (+ 315us) server_negotiation.cc:407] Connection header received
0929 20:53:39.530158 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:39.530163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:39.530293 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:39.530401 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:39.531382 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.532466 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.534295 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.534878 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.535011 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:39.537147 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:39.537154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:39.537158 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:39.537207 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:39.540418 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:39.541212 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:39.541218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:39.541222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:39.541377 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:39.541787 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:39.541792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:39.541795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:39.542214 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:39.542380 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:39.542767 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 20:53:39.543121 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 20:53:39.544735 6536 tablet_copy_service.cc:140] P 347ec01fc4094722a63b6055158d723c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 60d848cea40b4d6e9c79fdda2fa9c401 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:51864)
I20260929 20:53:39.545032 6536 tablet_copy_service.cc:161] P 347ec01fc4094722a63b6055158d723c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 60d848cea40b4d6e9c79fdda2fa9c401 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:51864: session id = 60d848cea40b4d6e9c79fdda2fa9c401-00000000000000000000000000000000
I20260929 20:53:39.549213 6536 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 20:53:39.551386 6977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 20:53:39.558238 6977 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: tablet copy: Starting download of 0 data blocks...
I20260929 20:53:39.558547 6977 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: tablet copy: Starting download of 1 WAL segments...
I20260929 20:53:39.561321 6977 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 20:53:39.565680 6536 tablet_copy_service.cc:342] P 347ec01fc4094722a63b6055158d723c: Request end of tablet copy session 60d848cea40b4d6e9c79fdda2fa9c401-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:51864
I20260929 20:53:39.565884 6536 tablet_copy_service.cc:433] P 347ec01fc4094722a63b6055158d723c: ending tablet copy session 60d848cea40b4d6e9c79fdda2fa9c401-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 60d848cea40b4d6e9c79fdda2fa9c401
I20260929 20:53:39.577431 6977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:39.579921 7088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:39.580417 7089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:39.587755 7091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:39.588250 6977 server_base.cc:1034] running on GCE node
I20260929 20:53:39.588581 6977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:39.589227 6977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:39.590444 6977 hybrid_clock.cc:648] HybridClock initialized: now 1790715219590367 us; error 50 us; skew 500 ppm
I20260929 20:53:39.590653 6977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:39 dist-test-slave-m9g1 krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715219, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:39.592711 6977 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260929 20:53:39.593946 6977 webserver.cc:460] Webserver started at http://127.0.97.61:46371/ using document root <none> and password file <none>
I20260929 20:53:39.594307 6977 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:39.594384 6977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:39.598898 6977 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 20:53:39.600850 7098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:39.601358 6977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260929 20:53:39.601487 6977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/wal
uuid: "60d848cea40b4d6e9c79fdda2fa9c401"
format_stamp: "Formatted at 2026-09-29 20:53:39 on dist-test-slave-m9g1"
server_key: "678639fb6f6c55ea404e5765377217ae"
server_key_iv: "82fd7e3b3bc14240a68a4f55f5b793cb"
server_key_version: "encryptionkey@0"
I20260929 20:53:39.601655 6977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:39.646860 6977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:39.650832 6977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:39.668980 6977 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40257
I20260929 20:53:39.669036 7149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40257 every 8 connection(s)
I20260929 20:53:39.670099 6977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-1/data/info.pb
I20260929 20:53:39.675101 7150 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:53:39.678856 7150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: Bootstrap starting.
I20260929 20:53:39.699940 7150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:53:39.700585 7150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: Bootstrap complete.
I20260929 20:53:39.701782 7150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: true } }
I20260929 20:53:39.702062 7150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 60d848cea40b4d6e9c79fdda2fa9c401, State: Initialized, Role: LEARNER
I20260929 20:53:39.702328 7150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: true } }
I20260929 20:53:39.703436 7150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:53:39.703236 7153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: true } } }
I20260929 20:53:39.703904 7153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [sys.catalog]: This master's current role is: LEARNER
I20260929 20:53:39.716897 7164 catalog_manager.cc:1259] Loaded cluster ID: 4418e57d94d541228f2db15ec1ae6260
I20260929 20:53:39.717002 7164 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: loading cluster ID for follower catalog manager: success
I20260929 20:53:39.719609 7164 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: acquiring CA information for follower catalog manager: success
I20260929 20:53:39.721367 7164 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 20:53:39.873427 7077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:39.873711 7077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:39.873806 7077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:39.880851 7077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:53:39.880951 7077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:39.880996 7077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:39.881050 7077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:53:39.881093 7077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:53:39.890895 7077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34335,127.0.97.61:40257,127.0.97.60:41713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:39.892004 7077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:39.893149 7077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:39.905260 7169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:39.905226 7170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:39.905195 7172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:39.906399 7077 server_base.cc:1034] running on GCE node
I20260929 20:53:39.907123 7077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:39.908496 7077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:39.909686 7077 hybrid_clock.cc:648] HybridClock initialized: now 1790715219909642 us; error 46 us; skew 500 ppm
I20260929 20:53:39.910000 7077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:53:39.911615 6972 heartbeater.cc:503] Master 127.0.97.62:34335 was elected leader, sending a full tablet report...
Sep 29 20:53:39 dist-test-slave-m9g1 krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715219, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:39.913297 7077 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260929 20:53:39.914693 7077 webserver.cc:460] Webserver started at http://127.0.97.60:41417/ using document root <none> and password file <none>
I20260929 20:53:39.915323 7077 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:39.915432 7077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:39.915705 7077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:39.918638 7077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/data/instance:
uuid: "42034594181341f4b03369c668783273"
format_stamp: "Formatted at 2026-09-29 20:53:39 on dist-test-slave-m9g1"
server_key: "b7d5c3822b9ea3f188f5e0fab4717231"
server_key_iv: "99983125e69996f0ae9314f29d625af4"
server_key_version: "encryptionkey@0"
I20260929 20:53:39.919337 7077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/wal/instance:
uuid: "42034594181341f4b03369c668783273"
format_stamp: "Formatted at 2026-09-29 20:53:39 on dist-test-slave-m9g1"
server_key: "b7d5c3822b9ea3f188f5e0fab4717231"
server_key_iv: "99983125e69996f0ae9314f29d625af4"
server_key_version: "encryptionkey@0"
I20260929 20:53:39.924665 7077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:53:39.928695 7180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:39.928951 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:39.922659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40257 (local address 127.0.0.1:60926)
0929 20:53:39.922813 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:39.922828 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:39.922916 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:39.923722 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:39.923726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:39.923753 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:39.923881 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:39.923889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.925157 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.925162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:39.927055 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:39.927071 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.928339 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.928346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:39.928499 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:39.928540 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 20:53:39.928616 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 20:53:39.928715 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 20:53:39.929219 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:39.922841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60926 (local address 127.0.97.61:40257)
0929 20:53:39.923363 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:39.923368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:39.923392 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:53:39.923460 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:39.923465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:39.923557 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:39.923670 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:39.924026 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.925014 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.927184 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.928168 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.928293 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:39.928527 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:39.928652 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 20:53:39.928775 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":268,"threads_started":1}
I20260929 20:53:39.930063 7125 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 20:53:39.930387 7077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:53:39.930546 7077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/wal
uuid: "42034594181341f4b03369c668783273"
format_stamp: "Formatted at 2026-09-29 20:53:39 on dist-test-slave-m9g1"
server_key: "b7d5c3822b9ea3f188f5e0fab4717231"
server_key_iv: "99983125e69996f0ae9314f29d625af4"
server_key_version: "encryptionkey@0"
I20260929 20:53:39.930550 7153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 347ec01fc4094722a63b6055158d723c. Latest consensus state: current_term: 1 leader_uuid: "347ec01fc4094722a63b6055158d723c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: true } } }
I20260929 20:53:39.930765 7153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [sys.catalog]: This master's current role is: LEARNER
I20260929 20:53:39.930739 7077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:39.958493 7077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:39.962598 7077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:39.962847 7077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:39.980477 7077 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41713
I20260929 20:53:39.980479 7231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41713 every 8 connection(s)
I20260929 20:53:39.981866 7077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/data/info.pb
I20260929 20:53:39.985610 7232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:39.990445 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 7077
I20260929 20:53:39.991767 7232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:53:39 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715219, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 29 20:53:39 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 20:53:40.013209 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:39.994390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.0.1:55834)
0929 20:53:39.995013 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:39.995059 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:39.995389 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:39.996303 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:39.996313 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:39.997090 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:39.997581 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:39.997603 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.999046 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:39.999051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.000844 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.000863 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.001966 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.001974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.002177 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.002844 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:40.002897 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:40.005537 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:40.008483 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.008497 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.008516 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.009236 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.009631 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.009636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.009643 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.009865 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.010475 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:40.010482 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:40.010991 (+ 509us) client_negotiation.cc:769] Sending connection context
0929 20:53:40.011289 (+ 298us) client_negotiation.cc:240] Negotiation successful
0929 20:53:40.011810 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":245,"threads_started":1}
I20260929 20:53:40.013229 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:39.992442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49636 (local address 127.0.97.60:41713)
0929 20:53:39.993153 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:39.993168 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:39.993213 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:53:39.995237 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:39.995257 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:39.995488 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:39.995681 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:39.997743 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.999336 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.001027 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.001902 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.002077 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.006467 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:40.006475 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:40.006488 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:40.006519 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:40.009840 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.010644 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.010651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.010654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.010799 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.011147 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.011151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.011154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.011590 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:40.011867 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:40.012170 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 20:53:40.012414 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":316,"threads_started":1}
I20260929 20:53:40.014315 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:39.994530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55834 (local address 127.0.97.62:34335)
0929 20:53:39.994699 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:39.994705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:39.995155 (+ 450us) server_negotiation.cc:407] Connection header received
0929 20:53:39.995602 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:39.995608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:39.995680 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:39.995768 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:39.997741 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:39.998925 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.001023 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.001719 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.001887 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.005707 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:40.005712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:40.005715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:40.005747 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:40.008323 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.009360 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.009365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.009367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.009524 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.009964 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.009969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.009972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.010339 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:40.010485 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:40.013750 (+ 3265us) server_negotiation.cc:299] Negotiation successful
0929 20:53:40.013994 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:53:40.015542 7232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:53:40 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715219, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 20:53:40.026543 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.016361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40257 (local address 127.0.0.1:60928)
0929 20:53:40.016542 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:40.016557 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:40.016632 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:40.016874 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:40.016876 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:40.017211 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:40.017334 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.017342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.018248 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.018252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.019148 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.019158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.019740 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.019744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.019869 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.020327 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:40.020342 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:40.022104 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:40.024729 (+ 2625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.024733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.024735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.025083 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.025370 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.025373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.025375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.025483 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.025868 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:40.025871 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:40.025994 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 20:53:40.026093 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 20:53:40.026315 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 20:53:40.026568 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.016392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60928 (local address 127.0.97.61:40257)
0929 20:53:40.016578 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:40.016583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:40.016600 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:53:40.016717 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:40.016721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:40.016782 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:40.016853 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:40.017434 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.018124 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.019249 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.019642 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.019729 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.022333 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:40.022338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:40.022341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:40.022371 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:40.024616 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.025169 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.025173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.025175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.025294 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.025552 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.025555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.025557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.025794 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:40.025902 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:40.026159 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 20:53:40.026353 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:53:40.027534 7232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:53:40 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715219, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 20:53:40.039022 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.028186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41713 (local address 127.0.0.1:49650)
0929 20:53:40.028341 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:40.028359 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:40.028443 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:40.028702 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:40.028705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:40.028993 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:40.029124 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.029131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.030036 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.030040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.030881 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.030892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.031513 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.031517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.031679 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.032124 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:40.032138 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:40.034461 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:40.036931 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.036936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.036940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.037240 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.037547 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.037552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.037556 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.037681 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.038122 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:40.038126 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:40.038276 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 20:53:40.038383 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 20:53:40.038628 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 20:53:40.039029 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.028374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49650 (local address 127.0.97.60:41713)
0929 20:53:40.028514 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:40.028518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:40.028531 (+ 13us) server_negotiation.cc:407] Connection header received
0929 20:53:40.028565 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:40.028568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:40.028617 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:40.028685 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:40.029253 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.029916 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.031007 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.031386 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.031487 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.034638 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:40.034645 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:40.034650 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:40.034681 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:40.036819 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.037346 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.037351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.037354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.037455 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.037772 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.037777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.037781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.038022 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:40.038138 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:40.038476 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 20:53:40.038747 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":54}
I20260929 20:53:40.043776 7232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: Bootstrap starting.
I20260929 20:53:40.046825 7232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:40.047756 7232 log.cc:824] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:40.050262 7232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: No bootstrap required, opened a new log
I20260929 20:53:40.057391 7232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } }
I20260929 20:53:40.057668 7232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:40.057745 7232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42034594181341f4b03369c668783273, State: Initialized, Role: FOLLOWER
I20260929 20:53:40.058287 7232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } }
I20260929 20:53:40.059017 7238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } } }
I20260929 20:53:40.059309 7238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:53:40.059921 7232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 20:53:40.068593 7249 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 20:53:40.068704 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 6577
W20260929 20:53:40.068790 7249 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:53:40.073530 7077 master_runner.cc:428] Detected that this master 42034594181341f4b03369c668783273 is joining an existing cluster
I20260929 20:53:40.073645 7077 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:41713
Sep 29 20:53:40 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260929 20:53:40.090601 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.069750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44432 (local address 127.0.97.1:46367)
0929 20:53:40.070448 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:40.070457 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:40.070485 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:53:40.071052 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:40.071063 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:40.071319 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:40.071527 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:40.072652 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.074177 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.075984 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.076783 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.076914 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.082143 (+ 5229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:40.082157 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:40.082173 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:40.082237 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:40.086677 (+ 4440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.087511 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.087519 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.087522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.087673 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.088077 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.088082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.088085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.088513 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:40.088786 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:40.089940 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0929 20:53:40.090178 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":353,"threads_started":1}
I20260929 20:53:40.091948 6700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:40.099099 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.077520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55846 (local address 127.0.97.62:34335)
0929 20:53:40.079412 (+ 1892us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:40.079418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:40.079438 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:53:40.079493 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:40.079498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:40.079574 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:40.079678 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:40.080914 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.081995 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.084112 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.084740 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.084833 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.087577 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:40.087583 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:40.087587 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:40.087625 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:40.090635 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.096861 (+ 6226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.096879 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.096882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.097034 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.097485 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.097490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.097493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.097793 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:40.097941 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:40.098574 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 20:53:40.098781 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1756}
I20260929 20:53:40.100024 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.077462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.0.1:55846)
0929 20:53:40.078555 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:40.078574 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:40.079208 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:40.079720 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:40.079725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:40.080377 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:40.080719 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.080732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.082285 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.082290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.083946 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.083960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.085116 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.085123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.085294 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.086324 (+ 1030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:40.086347 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:40.087395 (+ 1048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:40.091055 (+ 3660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.091059 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.091062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.091502 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.097170 (+ 5668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.097174 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.097177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.097364 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.098016 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:40.098020 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:40.098240 (+ 220us) client_negotiation.cc:769] Sending connection context
0929 20:53:40.099261 (+ 1021us) client_negotiation.cc:240] Negotiation successful
0929 20:53:40.099557 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"thread_start_us":2829,"threads_started":1}
I20260929 20:53:40.106523 7238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:53:40.106829 7238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } }
I20260929 20:53:40.111913 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.085369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60944 (local address 127.0.97.61:40257)
0929 20:53:40.085542 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:40.085547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:40.092184 (+ 6637us) server_negotiation.cc:407] Connection header received
0929 20:53:40.092442 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:40.092448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:40.092527 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:40.092593 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:40.094102 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.095230 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.101230 (+ 6000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.101849 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.101957 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.105402 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:40.105408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:40.105412 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:40.105457 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:40.108854 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.109747 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.109752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.109755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.109892 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.110295 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.110300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.110303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.110622 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:40.110745 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:40.111346 (+ 601us) server_negotiation.cc:299] Negotiation successful
0929 20:53:40.111566 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 20:53:40.112887 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.080115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49664 (local address 127.0.97.60:41713)
0929 20:53:40.080292 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:40.080298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:40.091582 (+ 11284us) server_negotiation.cc:407] Connection header received
0929 20:53:40.092001 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:40.092007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:40.092070 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:40.092133 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:40.093395 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.094487 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.097164 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.097854 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.097929 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.103753 (+ 5824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:40.103758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:40.103762 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:40.103795 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:40.108862 (+ 5067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.109729 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.109734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.109737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.109877 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.110293 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.110297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.110300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.110623 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:40.110756 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:40.112163 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0929 20:53:40.112455 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:53:40.111948 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.078549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41713 (local address 127.0.0.1:49664)
0929 20:53:40.091498 (+ 12949us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:40.091517 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:40.091856 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:40.092298 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:40.092303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:40.092887 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:40.093229 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.093240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.094685 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.094690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.097042 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.097053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.099046 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.099052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.099197 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.099883 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:40.099904 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:40.103588 (+ 3684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:40.108992 (+ 5404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.108996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.109001 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.109613 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.109987 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.109991 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.109995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.110157 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.110809 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:40.110814 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:40.111074 (+ 260us) client_negotiation.cc:769] Sending connection context
0929 20:53:40.111214 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 20:53:40.111500 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12822,"mutex_wait_us":218,"spinlock_wait_cycles":36224,"thread_start_us":13033,"threads_started":1}
I20260929 20:53:40.113443 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.086017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40257 (local address 127.0.0.1:60944)
0929 20:53:40.092094 (+ 6077us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:40.092110 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:40.092284 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:40.092786 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:40.092791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:40.093552 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:40.093915 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.093927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.095439 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.095444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.101067 (+ 5623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.101079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.102265 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.102271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.102445 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.103967 (+ 1522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:40.103993 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:40.105191 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:40.108992 (+ 3801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.108996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.109001 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.109638 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.110008 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.110012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.110015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.110173 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.110810 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:40.110814 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:40.111082 (+ 268us) client_negotiation.cc:769] Sending connection context
0929 20:53:40.112240 (+ 1158us) client_negotiation.cc:240] Negotiation successful
0929 20:53:40.112621 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5957,"mutex_wait_us":66,"spinlock_wait_cycles":26240,"thread_start_us":10544,"threads_started":1}
I20260929 20:53:40.124881 7238 leader_election.cc:288] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 347ec01fc4094722a63b6055158d723c (127.0.97.62:34335), 60d848cea40b4d6e9c79fdda2fa9c401 (127.0.97.61:40257)
I20260929 20:53:40.125694 7125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42034594181341f4b03369c668783273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" is_pre_election: true
I20260929 20:53:40.127512 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42034594181341f4b03369c668783273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "347ec01fc4094722a63b6055158d723c" is_pre_election: true
I20260929 20:53:40.128461 7183 leader_election.cc:302] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42034594181341f4b03369c668783273; no voters: 347ec01fc4094722a63b6055158d723c, 60d848cea40b4d6e9c79fdda2fa9c401
I20260929 20:53:40.128931 7238 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 29 20:53:40 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715217, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 20:53:40.141084 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.123766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41713 (local address 127.0.0.1:49672)
0929 20:53:40.124134 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:40.124151 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:40.124259 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:40.124987 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:40.124993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:40.125415 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:40.125622 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.125634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.127298 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.127304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.128651 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.128665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.130032 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.130039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.130237 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.130921 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:40.130944 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:40.133573 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:40.138638 (+ 5065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.138644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.138648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.139103 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.139457 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.139462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.139465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.139610 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.140171 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:40.140176 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:40.140326 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 20:53:40.140475 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 20:53:40.140728 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260929 20:53:40.141374 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.124461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49672 (local address 127.0.97.60:41713)
0929 20:53:40.124619 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:40.124624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:40.124640 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:53:40.124739 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:40.124744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:40.124821 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:40.125764 (+ 943us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:40.125917 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.127127 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.129195 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.129864 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.130575 (+ 711us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.133802 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:40.133808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:40.133813 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:40.133846 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:40.138507 (+ 4661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.139221 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.139227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.139230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.139356 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.139710 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.139715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.139719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.140059 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:40.140199 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:40.140579 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 20:53:40.140798 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:53:40.142858 6516 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:41713: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "42034594181341f4b03369c668783273"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 41713
}
attrs {
promote: true
}
}
dest_uuid: "347ec01fc4094722a63b6055158d723c"
cas_config_opid_index: 5
I20260929 20:53:40.143834 6516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: true } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: true } }
I20260929 20:53:40.146484 7125 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 1 LEARNER]: Refusing update from remote peer 347ec01fc4094722a63b6055158d723c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 20:53:40.145547 7075 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 42034594181341f4b03369c668783273 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: true } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: true } } }
I20260929 20:53:40.149919 7075 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:41713
I20260929 20:53:40.150491 7077 master.cc:537] Master@127.0.97.60:41713 shutting down...
I20260929 20:53:40.162150 7077 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:40.162585 7077 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:40.162786 7077 tablet_replica.cc:330] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: stopping tablet replica
I20260929 20:53:40.173116 7077 master.cc:552] Master@127.0.97.60:41713 shutdown complete.
I20260929 20:53:40.173218 7077 master_runner.cc:305] Clearing existing system tablet
I20260929 20:53:40.173617 7077 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:40.173704 7077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260929 20:53:40.174808 6500 consensus_peers.cc:487] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c -> Peer 42034594181341f4b03369c668783273 (127.0.97.60:41713): Couldn't send request to peer 42034594181341f4b03369c668783273. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260929 20:53:40.175400 7260 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 20:53:40.183205 7265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "347ec01fc4094722a63b6055158d723c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: true } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: true } } }
I20260929 20:53:40.183549 7265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:40.188884 7077 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.003s sys 0.000s
I20260929 20:53:40.188496 7125 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 42034594181341f4b03369c668783273 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: true } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: true } } }
I20260929 20:53:40.193234 7153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "347ec01fc4094722a63b6055158d723c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: true } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: true } } }
I20260929 20:53:40.193504 7153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [sys.catalog]: This master's current role is: LEARNER
I20260929 20:53:40.194620 7261 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c: attempting to promote NON_VOTER 60d848cea40b4d6e9c79fdda2fa9c401 to VOTER
I20260929 20:53:40.195443 7261 consensus_queue.cc:237] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: true } }
I20260929 20:53:40.197758 7125 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 1 LEARNER]: Refusing update from remote peer 347ec01fc4094722a63b6055158d723c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 20:53:40.197798 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.197323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41713 (local address 127.0.0.1:49686)
0929 20:53:40.197542 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:40.197627 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260929 20:53:40.198464 7267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:40.198428 7075 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 20:53:40.198923 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.198482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41713 (local address 127.0.0.1:49696)
0929 20:53:40.198662 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:40.198741 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260929 20:53:40.199450 6500 consensus_peers.cc:487] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c -> Peer 42034594181341f4b03369c668783273 (127.0.97.60:41713): Couldn't send request to peer 42034594181341f4b03369c668783273. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41713: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 20:53:40.200461 7077 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:40.200840 7077 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.006s
I20260929 20:53:40.200982 7077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/wal
uuid: "42034594181341f4b03369c668783273"
format_stamp: "Formatted at 2026-09-29 20:53:39 on dist-test-slave-m9g1"
server_key: "b7d5c3822b9ea3f188f5e0fab4717231"
server_key_iv: "99983125e69996f0ae9314f29d625af4"
server_key_version: "encryptionkey@0"
I20260929 20:53:40.200984 7260 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 60d848cea40b4d6e9c79fdda2fa9c401 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: true } } }
I20260929 20:53:40.205008 7075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "347ec01fc4094722a63b6055158d723c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: true } } }
I20260929 20:53:40.205250 7075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:40.205606 7260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "347ec01fc4094722a63b6055158d723c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: true } } }
I20260929 20:53:40.205953 7260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:40.206243 7077 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:40.207154 7124 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 60d848cea40b4d6e9c79fdda2fa9c401 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: true } } }
I20260929 20:53:40.210872 7153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "347ec01fc4094722a63b6055158d723c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: true } } }
I20260929 20:53:40.211102 7153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:53:40.213764 7077 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 20:53:40.213945 7077 log.cc:1196] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 20:53:40.214480 7077 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: Deleting consensus metadata
I20260929 20:53:40.214745 7077 master_runner.cc:315] Copying system tablet from 127.0.97.62:34335
I20260929 20:53:40.223664 7077 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:34335
I20260929 20:53:40.238709 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.224559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55848 (local address 127.0.97.62:34335)
0929 20:53:40.224767 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:40.224773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:40.225376 (+ 603us) server_negotiation.cc:407] Connection header received
0929 20:53:40.225567 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:40.225573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:40.225653 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:40.225750 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:40.226807 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.227975 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.229813 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.230474 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.230592 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.232766 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:40.232772 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:40.232777 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:40.232815 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:40.235985 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.236769 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.236774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.236778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.236925 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.237289 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.237294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.237297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.237606 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:40.237748 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:40.238130 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 20:53:40.238365 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 20:53:40.238777 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.224747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.0.1:55848)
0929 20:53:40.225259 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:40.225278 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:40.225426 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:40.225791 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:40.225796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:40.226382 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:40.226644 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.226656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.228135 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.228141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.229654 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.229669 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.230620 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.230626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.230817 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.231497 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:40.231520 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:40.232569 (+ 1049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:40.236101 (+ 3532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.236107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.236111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.236592 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.237028 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.237033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.237037 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.237197 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.237716 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:40.237721 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:40.237891 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 20:53:40.238039 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 20:53:40.238359 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":247,"threads_started":1}
I20260929 20:53:40.239652 6536 tablet_copy_service.cc:140] P 347ec01fc4094722a63b6055158d723c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 42034594181341f4b03369c668783273 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:55848)
I20260929 20:53:40.239813 6536 tablet_copy_service.cc:161] P 347ec01fc4094722a63b6055158d723c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 42034594181341f4b03369c668783273 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:55848: session id = 42034594181341f4b03369c668783273-00000000000000000000000000000000
I20260929 20:53:40.243685 6536 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 20:53:40.245414 7077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 20:53:40.251922 7077 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: tablet copy: Starting download of 0 data blocks...
I20260929 20:53:40.252185 7077 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: tablet copy: Starting download of 1 WAL segments...
I20260929 20:53:40.254733 7077 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 20:53:40.258860 6536 tablet_copy_service.cc:342] P 347ec01fc4094722a63b6055158d723c: Request end of tablet copy session 42034594181341f4b03369c668783273-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:55848
I20260929 20:53:40.258988 6536 tablet_copy_service.cc:433] P 347ec01fc4094722a63b6055158d723c: ending tablet copy session 42034594181341f4b03369c668783273-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 42034594181341f4b03369c668783273
I20260929 20:53:40.269678 7077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:40.272115 7274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:40.272727 7275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:40.274681 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 6577
W20260929 20:53:40.292012 7277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:40.292338 7077 server_base.cc:1034] running on GCE node
I20260929 20:53:40.292675 7077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:40.293501 7077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:40.294688 7077 hybrid_clock.cc:648] HybridClock initialized: now 1790715220294639 us; error 38 us; skew 500 ppm
I20260929 20:53:40.294939 7077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:40 dist-test-slave-m9g1 krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:40.296893 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:46367
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34335
--webserver_port=44769
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:34335,127.0.97.61:40257
--tserver_master_addrs=127.0.97.62:34335,127.0.97.61:40257,127.0.97.60:41713 with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:53:40.297029 7077 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260929 20:53:40.303012 7077 webserver.cc:460] Webserver started at http://127.0.97.60:42261/ using document root <none> and password file <none>
I20260929 20:53:40.303356 7077 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:40.303453 7077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:40.306692 7077 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 20:53:40.308589 7285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:40.308979 7077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260929 20:53:40.309092 7077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/wal
uuid: "42034594181341f4b03369c668783273"
format_stamp: "Formatted at 2026-09-29 20:53:39 on dist-test-slave-m9g1"
server_key: "b7d5c3822b9ea3f188f5e0fab4717231"
server_key_iv: "99983125e69996f0ae9314f29d625af4"
server_key_version: "encryptionkey@0"
I20260929 20:53:40.309265 7077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:40.327023 7077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:40.330782 7077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:40.356571 7077 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41713
I20260929 20:53:40.356769 7336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41713 every 8 connection(s)
I20260929 20:53:40.357725 7077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/master-2/data/info.pb
I20260929 20:53:40.385829 7337 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:53:40.388749 7337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: Bootstrap starting.
I20260929 20:53:40.402259 7337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:53:40.402669 7337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: Bootstrap complete.
I20260929 20:53:40.403559 7337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: true } }
I20260929 20:53:40.403831 7337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42034594181341f4b03369c668783273, State: Initialized, Role: LEARNER
I20260929 20:53:40.404088 7337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: true } }
I20260929 20:53:40.410807 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: true } } }
I20260929 20:53:40.411132 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [sys.catalog]: This master's current role is: LEARNER
I20260929 20:53:40.411737 7337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:53:40.423208 7351 catalog_manager.cc:1259] Loaded cluster ID: 4418e57d94d541228f2db15ec1ae6260
I20260929 20:53:40.423306 7351 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: loading cluster ID for follower catalog manager: success
I20260929 20:53:40.425580 7351 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: acquiring CA information for follower catalog manager: success
I20260929 20:53:40.427328 7351 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 20:53:40.660907 7282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:40.661164 7282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:40.661289 7282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:40.668289 7282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:40.668406 7282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:40.668552 7282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:53:40.677886 7282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:46367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44769
--tserver_master_addrs=127.0.97.62:34335,127.0.97.61:40257,127.0.97.60:41713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:40.679131 7282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:40.680382 7282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:40.693152 7356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:40.693197 7357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:40.693115 7359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:40.693873 7282 server_base.cc:1034] running on GCE node
I20260929 20:53:40.694470 7282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:40.695277 7282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:40.696455 7282 hybrid_clock.cc:648] HybridClock initialized: now 1790715220696401 us; error 48 us; skew 500 ppm
I20260929 20:53:40.696754 7282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:40 dist-test-slave-m9g1 krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:40.703274 7282 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 20:53:40.704694 7282 webserver.cc:460] Webserver started at http://127.0.97.1:44769/ using document root <none> and password file <none>
I20260929 20:53:40.705405 7282 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:40.705524 7282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:40.710624 7282 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:53:40.713840 7366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:40.715219 7282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:53:40.715376 7282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "e816c455c3944cac8973eaac1bf8580a"
format_stamp: "Formatted at 2026-09-29 20:53:37 on dist-test-slave-m9g1"
server_key: "e090b4df7ca38c0a7e02fbab26e545fa"
server_key_iv: "92523ead42251627220dab698208f63b"
server_key_version: "encryptionkey@0"
I20260929 20:53:40.716338 7282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:40.731580 7282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:40.735610 7282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:40.735863 7282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:40.736828 7282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:40.738098 7282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:40.738219 7282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:40.738334 7282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:40.738402 7282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:40.765345 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.757446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41713 (local address 127.0.0.1:49702)
0929 20:53:40.759028 (+ 1582us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:40.759048 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:40.759206 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:40.759613 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:40.759618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:40.759657 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:40.759834 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.759842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.761340 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.761347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.763158 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.763176 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.764620 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.764626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.764792 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.764831 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 20:53:40.764920 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 20:53:40.765047 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1471,"thread_start_us":1384,"threads_started":1}
I20260929 20:53:40.765358 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.757569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49702 (local address 127.0.97.60:41713)
0929 20:53:40.758706 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:40.758712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:40.759129 (+ 417us) server_negotiation.cc:407] Connection header received
0929 20:53:40.759353 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:40.759359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:40.759431 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:40.759543 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:40.759968 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.761173 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.763313 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.764438 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.764563 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.764771 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:40.764954 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 20:53:40.765080 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":994,"thread_start_us":869,"threads_started":1}
I20260929 20:53:40.766644 7312 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 20:53:40.767307 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 347ec01fc4094722a63b6055158d723c. Latest consensus state: current_term: 1 leader_uuid: "347ec01fc4094722a63b6055158d723c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: true } } }
I20260929 20:53:40.767550 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [sys.catalog]: This master's current role is: LEARNER
I20260929 20:53:40.771169 7282 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46367
I20260929 20:53:40.771214 7481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46367 every 8 connection(s)
I20260929 20:53:40.772497 7282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:53:40.777309 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 7282
I20260929 20:53:40.778162 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 6711
Sep 29 20:53:40 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 29 20:53:40 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Sep 29 20:53:40 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 20:53:40 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 20:53:40.826604 7490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.776478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36133 (local address 127.0.97.62:34335)
0929 20:53:40.801144 (+ 24666us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:40.801152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:40.801180 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:53:40.805355 (+ 4175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:40.805362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:40.805469 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:40.805555 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:40.808273 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.809669 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.812689 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.813370 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.813465 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.819670 (+ 6205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:40.819676 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:40.819681 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:40.819718 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:40.823275 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.824128 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.824133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.824136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.824253 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.824783 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.824788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.824792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.825040 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:40.825165 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:40.825991 (+ 826us) server_negotiation.cc:299] Negotiation successful
0929 20:53:40.826216 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24514,"thread_start_us":27134,"threads_started":1}
I20260929 20:53:40.827586 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.775048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40347 (local address 127.0.97.61:40257)
0929 20:53:40.805346 (+ 30298us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:40.805353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:40.811148 (+ 5795us) server_negotiation.cc:407] Connection header received
0929 20:53:40.811270 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:40.811276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:40.811357 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:40.811446 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:40.813254 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.814364 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.816156 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.816540 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.816638 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.820952 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:40.820958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:40.820962 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:40.820996 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:40.824037 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.825193 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.825208 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.825212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.825329 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.825734 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.825739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.825742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.826336 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:40.826450 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:40.826994 (+ 544us) server_negotiation.cc:299] Negotiation successful
0929 20:53:40.827186 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30173,"thread_start_us":30078,"threads_started":1}
I20260929 20:53:40.828676 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.779504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.0.97.2:36383)
0929 20:53:40.805647 (+ 26143us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:40.805656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:40.806356 (+ 700us) server_negotiation.cc:407] Connection header received
0929 20:53:40.806615 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:40.806625 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:40.806904 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:40.807069 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:40.809304 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.811615 (+ 2311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.813430 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.814243 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.814347 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.820762 (+ 6415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:40.820774 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:40.820790 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:40.820830 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:40.825049 (+ 4219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.825811 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.825819 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.825822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.825950 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.826422 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.826427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.826431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.826817 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:40.827059 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:40.828076 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0929 20:53:40.828319 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26000,"thread_start_us":25859,"threads_started":1}
I20260929 20:53:40.829839 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.775638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.1:36133)
0929 20:53:40.799634 (+ 23996us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:40.799703 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:40.804927 (+ 5224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:40.806267 (+ 1340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:40.806283 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:40.807141 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:40.808059 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.808085 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.809894 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.809900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.812530 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.812546 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.813928 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.813936 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.814265 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.815734 (+ 1469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:40.815759 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:40.819402 (+ 3643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:40.823439 (+ 4037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.823448 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.823465 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.824008 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.824383 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.824387 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.824391 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.824651 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.825161 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:40.825168 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:40.825569 (+ 401us) client_negotiation.cc:769] Sending connection context
0929 20:53:40.827171 (+ 1602us) client_negotiation.cc:240] Negotiation successful
0929 20:53:40.828107 (+ 936us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23573,"thread_start_us":25460,"threads_started":1}
I20260929 20:53:40.829959 6834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:40.829895 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.775638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40257 (local address 127.0.97.1:40347)
0929 20:53:40.811047 (+ 35409us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:40.811062 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:40.811168 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:40.811639 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:40.811643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:40.812323 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:40.813043 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.813057 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.814518 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.814522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.816023 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.816039 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.816716 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.816727 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.816960 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.817497 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:40.817518 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:40.820762 (+ 3244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:40.824173 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.824178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.824182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.824687 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.825457 (+ 770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.825461 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.825465 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.825614 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.826538 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:40.826542 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:40.826671 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 20:53:40.827802 (+ 1131us) client_negotiation.cc:240] Negotiation successful
0929 20:53:40.828259 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35267,"mutex_wait_us":241,"spinlock_wait_cycles":108800,"thread_start_us":39317,"threads_started":1}
I20260929 20:53:40.831477 7483 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34335
I20260929 20:53:40.831765 7484 heartbeater.cc:347] Connected to a master server at 127.0.97.61:40257
I20260929 20:53:40.831884 7484 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:40.832167 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.776292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48525 (local address 127.0.97.60:41713)
0929 20:53:40.776471 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:40.776478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:40.815498 (+ 39020us) server_negotiation.cc:407] Connection header received
0929 20:53:40.815602 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:40.815607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:40.815686 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:40.815784 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:40.817001 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.818023 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.819902 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.820502 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.820595 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.823999 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:40.824005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:40.824009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:40.824045 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:40.827537 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.828511 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.828517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.828520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.828645 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:40.830088 (+ 1443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:40.830093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:40.830096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:40.830458 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:40.830574 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:40.831029 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 20:53:40.831814 (+ 785us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 20:53:40.832672 7484 heartbeater.cc:511] Master 127.0.97.61:40257 requested a full tablet report, sending...
I20260929 20:53:40.833130 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:40.775758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41713 (local address 127.0.97.1:48525)
0929 20:53:40.815387 (+ 39629us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:40.815405 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:40.815510 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:40.815957 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:40.815962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:40.816519 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:40.816822 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.816834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.818226 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.818230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.819734 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:40.819745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:40.820846 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:40.820852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:40.821029 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:40.821516 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:40.821536 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:40.823804 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:40.827723 (+ 3919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.827728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.827731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.828388 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.829401 (+ 1013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:40.829406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:40.829410 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:40.829959 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:40.830647 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:40.830651 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:40.830775 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 20:53:40.832400 (+ 1625us) client_negotiation.cc:240] Negotiation successful
0929 20:53:40.832704 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39509,"mutex_wait_us":58,"spinlock_wait_cycles":18304,"thread_start_us":34942,"threads_started":1}
I20260929 20:53:40.831779 7483 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:40.834100 7483 heartbeater.cc:511] Master 127.0.97.62:34335 requested a full tablet report, sending...
I20260929 20:53:40.835105 6516 ts_manager.cc:194] Re-registered known tserver with Master: e816c455c3944cac8973eaac1bf8580a (127.0.97.1:46367)
I20260929 20:53:40.835083 7115 ts_manager.cc:194] Registered new tserver with Master: e816c455c3944cac8973eaac1bf8580a (127.0.97.1:46367)
I20260929 20:53:40.835588 7482 heartbeater.cc:347] Connected to a master server at 127.0.97.60:41713
I20260929 20:53:40.835718 7482 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:40.836087 6516 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36133
I20260929 20:53:40.836148 7482 heartbeater.cc:511] Master 127.0.97.60:41713 requested a full tablet report, sending...
I20260929 20:53:40.837802 7302 ts_manager.cc:194] Registered new tserver with Master: e816c455c3944cac8973eaac1bf8580a (127.0.97.1:46367)
I20260929 20:53:40.919979 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 6711
I20260929 20:53:40.933463 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36383
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34335
--webserver_port=37373
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:34335,127.0.97.61:40257
--tserver_master_addrs=127.0.97.62:34335,127.0.97.61:40257,127.0.97.60:41713 with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:53:41.149628 7438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "347ec01fc4094722a63b6055158d723c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: true } } }
I20260929 20:53:41.150013 7438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:41.151386 7495 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c: attempting to promote NON_VOTER 42034594181341f4b03369c668783273 to VOTER
I20260929 20:53:41.151978 7495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: false } }
I20260929 20:53:41.153671 7312 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 1 LEARNER]: Refusing update from remote peer 347ec01fc4094722a63b6055158d723c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 20:53:41.153712 7124 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 1 FOLLOWER]: Refusing update from remote peer 347ec01fc4094722a63b6055158d723c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 20:53:41.154266 7496 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 20:53:41.154714 7495 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 20:53:41.157660 7124 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 42034594181341f4b03369c668783273 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: false } } }
I20260929 20:53:41.157001 7496 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 42034594181341f4b03369c668783273 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: false } } }
I20260929 20:53:41.161653 7496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "347ec01fc4094722a63b6055158d723c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: false } } }
I20260929 20:53:41.161998 7496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:41.162365 7496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "347ec01fc4094722a63b6055158d723c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: false } } }
I20260929 20:53:41.162649 7496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:41.164382 7438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "347ec01fc4094722a63b6055158d723c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: false } } }
I20260929 20:53:41.164616 7438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:41.165504 7312 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 42034594181341f4b03369c668783273 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: false } } }
I20260929 20:53:41.169884 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "347ec01fc4094722a63b6055158d723c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: false } } }
I20260929 20:53:41.170121 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:53:41.170419 7499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "347ec01fc4094722a63b6055158d723c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: false } } }
I20260929 20:53:41.170815 7499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 20:53:41.245657 7494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:41.245927 7494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:41.246024 7494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:41.253237 7494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:41.253329 7494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:41.253445 7494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:53:41.262634 7494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37373
--tserver_master_addrs=127.0.97.62:34335,127.0.97.61:40257,127.0.97.60:41713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:41.263784 7494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:41.264954 7494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:41.276764 7507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:41.276743 7505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:41.277184 7504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:41.277631 7494 server_base.cc:1034] running on GCE node
I20260929 20:53:41.278258 7494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:41.279084 7494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:41.280323 7494 hybrid_clock.cc:648] HybridClock initialized: now 1790715221280258 us; error 45 us; skew 500 ppm
I20260929 20:53:41.280674 7494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:41 dist-test-slave-m9g1 krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:41.283680 7494 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 20:53:41.284967 7494 webserver.cc:460] Webserver started at http://127.0.97.2:37373/ using document root <none> and password file <none>
I20260929 20:53:41.285585 7494 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:41.285706 7494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:41.290658 7494 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:53:41.293041 7514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:41.294261 7494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260929 20:53:41.294391 7494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "9f3bebe27bd744c1bcd36918f642ca91"
format_stamp: "Formatted at 2026-09-29 20:53:38 on dist-test-slave-m9g1"
server_key: "b6c1915516a47486f8b9fd1bc0d318ac"
server_key_iv: "c1959e1310f0031b8fbc5e65abeaf223"
server_key_version: "encryptionkey@0"
I20260929 20:53:41.295255 7494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:41.315414 7494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:41.319257 7494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:41.319505 7494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:41.320395 7494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:41.321669 7494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:41.321758 7494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:41.321861 7494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:41.321926 7494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:41.352914 7494 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36383
I20260929 20:53:41.352974 7626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36383 every 8 connection(s)
I20260929 20:53:41.354261 7494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:53:41.361057 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 7494
I20260929 20:53:41.361325 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 6844
Sep 29 20:53:41 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Sep 29 20:53:41 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 20:53:41 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 20:53:41 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:53:41.409142 7632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:41.357928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60411 (local address 127.0.97.60:41713)
0929 20:53:41.389260 (+ 31332us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:41.389267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:41.389285 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:53:41.389342 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:41.389348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:41.389424 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:41.389546 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:41.392019 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.393569 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.396309 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.396838 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.396941 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:41.401165 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:41.401172 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:41.401177 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:41.401220 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:41.405464 (+ 4244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:41.406426 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:41.406432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:41.406436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:41.406571 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:41.407070 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:41.407075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:41.407079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:41.407388 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:41.407507 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:41.408565 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0929 20:53:41.408763 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31186,"thread_start_us":31059,"threads_started":1}
I20260929 20:53:41.409384 7636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:41.362687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47603 (local address 127.0.97.61:40257)
0929 20:53:41.386022 (+ 23335us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:41.386029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:41.388485 (+ 2456us) server_negotiation.cc:407] Connection header received
0929 20:53:41.388564 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:41.388570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:41.388648 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:41.388739 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:41.392267 (+ 3528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.393314 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.396622 (+ 3308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.397296 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.397428 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:41.401578 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:41.401584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:41.401588 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:41.401643 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:41.405631 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:41.407327 (+ 1696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:41.407332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:41.407335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:41.407460 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:41.407843 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:41.407848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:41.407851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:41.408171 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:41.408295 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:41.408873 (+ 578us) server_negotiation.cc:299] Negotiation successful
0929 20:53:41.409074 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23211,"thread_start_us":23115,"threads_started":1}
I20260929 20:53:41.409898 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:41.362408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51738 (local address 127.0.97.3:45645)
0929 20:53:41.382987 (+ 20579us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:41.382997 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:41.383027 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:53:41.383100 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:41.383109 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:41.383389 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:41.383559 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:41.386564 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.390668 (+ 4104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.392412 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.393217 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.393328 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:41.401592 (+ 8264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:41.401610 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:41.401632 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:41.401680 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:41.406017 (+ 4337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:41.406779 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:41.406788 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:41.406791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:41.406933 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:41.407349 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:41.407354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:41.407358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:41.407763 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:41.408016 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:41.409095 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0929 20:53:41.409537 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20398,"thread_start_us":20263,"threads_started":1}
I20260929 20:53:41.410624 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:41.358278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40257 (local address 127.0.97.2:47603)
0929 20:53:41.388234 (+ 29956us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:41.388256 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:41.388382 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:41.390089 (+ 1707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:41.390094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:41.390720 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:41.391795 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:41.391822 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.393490 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.393496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:41.396485 (+ 2989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:41.396495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.397476 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.397483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:41.397658 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:41.398872 (+ 1214us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:41.398893 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:41.401343 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:41.405791 (+ 4448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:41.405795 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:41.405799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:41.407206 (+ 1407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:41.407580 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:41.407584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:41.407587 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:41.407731 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:41.408347 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:41.408351 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:41.408477 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 20:53:41.408622 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 20:53:41.409739 (+ 1117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29825,"mutex_wait_us":1561,"thread_start_us":29669,"threads_started":1}
I20260929 20:53:41.410789 7631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:41.366641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.2:52857)
0929 20:53:41.390060 (+ 23419us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:41.390075 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:41.390215 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:41.391195 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:41.391200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:41.391770 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:41.392076 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:41.392088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.393481 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.393486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:41.396485 (+ 2999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:41.396499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.398694 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.398700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:41.398896 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:41.399682 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:41.399704 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:41.401896 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:41.406254 (+ 4358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:41.406259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:41.406262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:41.406720 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:41.407277 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:41.407282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:41.407286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:41.407442 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:41.408059 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:41.408064 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:41.408195 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 20:53:41.408337 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 20:53:41.408805 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23310,"mutex_wait_us":559,"thread_start_us":5783,"threads_started":1}
I20260929 20:53:41.411193 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:41.366620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52857 (local address 127.0.97.62:34335)
0929 20:53:41.386444 (+ 19824us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:41.386451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:41.390163 (+ 3712us) server_negotiation.cc:407] Connection header received
0929 20:53:41.390848 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:41.390853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:41.390924 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:41.391034 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:41.392244 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.393321 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.397890 (+ 4569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.398488 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.398605 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:41.402692 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:41.402698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:41.402702 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:41.402742 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:41.406065 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:41.406828 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:41.406834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:41.406838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:41.406961 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:41.407568 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:41.407573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:41.407576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:41.407872 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:41.408001 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:41.409539 (+ 1538us) server_negotiation.cc:299] Negotiation successful
0929 20:53:41.409750 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19704,"thread_start_us":19931,"threads_started":1}
I20260929 20:53:41.411502 6967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:41.411981 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:41.357373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41713 (local address 127.0.97.2:60411)
0929 20:53:41.372827 (+ 15454us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:41.372871 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:41.374580 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:41.389874 (+ 15294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:41.389892 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:41.390726 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:41.391795 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:41.391823 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.393720 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.393727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:41.396179 (+ 2452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:41.396194 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.397055 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.397062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:41.397303 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:41.398092 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:41.398125 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:41.400803 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:41.405637 (+ 4834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:41.405645 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:41.405662 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:41.406296 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:41.406689 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:41.406694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:41.406697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:41.406924 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:41.407575 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:41.407583 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:41.407991 (+ 408us) client_negotiation.cc:769] Sending connection context
0929 20:53:41.408288 (+ 297us) client_negotiation.cc:240] Negotiation successful
0929 20:53:41.411607 (+ 3319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15161,"mutex_wait_us":3086,"spinlock_wait_cycles":18816,"thread_start_us":32317,"threads_started":1}
I20260929 20:53:41.412369 7628 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34335
I20260929 20:53:41.412453 7629 heartbeater.cc:347] Connected to a master server at 127.0.97.61:40257
I20260929 20:53:41.412664 7629 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:41.412671 7628 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:41.413591 7628 heartbeater.cc:511] Master 127.0.97.62:34335 requested a full tablet report, sending...
I20260929 20:53:41.413645 7627 heartbeater.cc:347] Connected to a master server at 127.0.97.60:41713
I20260929 20:53:41.413749 7627 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:41.414124 7627 heartbeater.cc:511] Master 127.0.97.60:41713 requested a full tablet report, sending...
I20260929 20:53:41.414539 7629 heartbeater.cc:511] Master 127.0.97.61:40257 requested a full tablet report, sending...
I20260929 20:53:41.415355 7302 ts_manager.cc:194] Registered new tserver with Master: 9f3bebe27bd744c1bcd36918f642ca91 (127.0.97.2:36383)
I20260929 20:53:41.415629 7115 ts_manager.cc:194] Registered new tserver with Master: 9f3bebe27bd744c1bcd36918f642ca91 (127.0.97.2:36383)
I20260929 20:53:41.415907 6516 ts_manager.cc:194] Re-registered known tserver with Master: 9f3bebe27bd744c1bcd36918f642ca91 (127.0.97.2:36383)
I20260929 20:53:41.416791 6516 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52857
I20260929 20:53:41.495987 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 6844
I20260929 20:53:41.509013 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45645
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34335
--webserver_port=45527
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:34335,127.0.97.61:40257
--tserver_master_addrs=127.0.97.62:34335,127.0.97.61:40257,127.0.97.60:41713 with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:41.815193 7640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:41.815443 7640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:41.815564 7640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:41.822476 7640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:41.822574 7640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:41.822708 7640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:53:41.831954 7640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45527
--tserver_master_addrs=127.0.97.62:34335,127.0.97.61:40257,127.0.97.60:41713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:41.833142 7640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:41.834470 7640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 20:53:41.838610 7483 heartbeater.cc:503] Master 127.0.97.62:34335 was elected leader, sending a full tablet report...
W20260929 20:53:41.847190 7645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:41.847159 7646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:41.847126 7648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:41.847815 7640 server_base.cc:1034] running on GCE node
I20260929 20:53:41.848409 7640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:41.849342 7640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:41.850524 7640 hybrid_clock.cc:648] HybridClock initialized: now 1790715221850475 us; error 39 us; skew 500 ppm
I20260929 20:53:41.850801 7640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:41 dist-test-slave-m9g1 krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:41.853590 7640 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 20:53:41.854898 7640 webserver.cc:460] Webserver started at http://127.0.97.3:45527/ using document root <none> and password file <none>
I20260929 20:53:41.855510 7640 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:41.855612 7640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:41.860612 7640 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 20:53:41.863057 7655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:41.864238 7640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 20:53:41.864367 7640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "3529304287894649b2750ee4aeb0fa05"
format_stamp: "Formatted at 2026-09-29 20:53:38 on dist-test-slave-m9g1"
server_key: "3aa9411c01ef247d029b0fcb1ab1a128"
server_key_iv: "7d0e3614b84cab17cfa9eee7cd4b1046"
server_key_version: "encryptionkey@0"
I20260929 20:53:41.865248 7640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:41.886432 7640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:41.889863 7640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:41.890087 7640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:41.890945 7640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:41.892171 7640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:41.892257 7640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:41.892364 7640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:41.892433 7640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:41.923652 7640 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45645
I20260929 20:53:41.923713 7767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45645 every 8 connection(s)
I20260929 20:53:41.924985 7640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:53:41.926384 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 7640
Sep 29 20:53:41 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 20:53:41 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:53:41.970714 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:41.928916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40257 (local address 127.0.97.3:59205)
0929 20:53:41.940317 (+ 11401us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:41.940336 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:41.940442 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:41.947726 (+ 7284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:41.947731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:41.948459 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:41.949402 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:41.949426 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.952085 (+ 2659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.952091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:41.953813 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:41.953827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.955022 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.955029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:41.955296 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:41.956121 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:41.956156 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:41.959246 (+ 3090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:41.963755 (+ 4509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:41.963764 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:41.963781 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:41.964332 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:41.966159 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:41.966165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:41.966169 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:41.966413 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:41.967844 (+ 1431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:41.967853 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:41.968250 (+ 397us) client_negotiation.cc:769] Sending connection context
0929 20:53:41.968554 (+ 304us) client_negotiation.cc:240] Negotiation successful
0929 20:53:41.969316 (+ 762us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11272,"mutex_wait_us":214,"thread_start_us":11122,"threads_started":1}
I20260929 20:53:41.972851 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:41.927754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59205 (local address 127.0.97.61:40257)
0929 20:53:41.947398 (+ 19644us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:41.947404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:41.947422 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:53:41.947473 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:41.947478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:41.947564 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:41.948159 (+ 595us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:41.950905 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.951910 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.953977 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.954648 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.954756 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:41.959501 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:41.959508 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:41.959513 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:41.959558 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:41.963600 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:41.965866 (+ 2266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:41.965871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:41.965874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:41.966028 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:41.967239 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:41.967244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:41.967248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:41.967641 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:41.967768 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:41.972244 (+ 4476us) server_negotiation.cc:299] Negotiation successful
0929 20:53:41.972482 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19513,"thread_start_us":19419,"threads_started":1}
I20260929 20:53:41.974294 7770 heartbeater.cc:347] Connected to a master server at 127.0.97.61:40257
I20260929 20:53:41.974617 7770 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:41.975479 7770 heartbeater.cc:511] Master 127.0.97.61:40257 requested a full tablet report, sending...
I20260929 20:53:41.977262 7115 ts_manager.cc:194] Registered new tserver with Master: 3529304287894649b2750ee4aeb0fa05 (127.0.97.3:45645)
I20260929 20:53:41.978250 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:41.928355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.3:52333)
0929 20:53:41.937001 (+ 8646us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:41.937064 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:41.939543 (+ 2479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:41.947147 (+ 7604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:41.947160 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:41.948259 (+ 1099us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:41.949399 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:41.949425 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.950889 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.950895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:41.955890 (+ 4995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:41.955903 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.961608 (+ 5705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.961615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:41.961793 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:41.964681 (+ 2888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:41.964703 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:41.967100 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:41.975583 (+ 8483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:41.975588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:41.975592 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:41.976072 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:41.976498 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:41.976504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:41.976508 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:41.976670 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:41.977250 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:41.977255 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:41.977391 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 20:53:41.977508 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 20:53:41.977771 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8302,"mutex_wait_us":2940,"spinlock_wait_cycles":18304,"thread_start_us":8072,"threads_started":1}
I20260929 20:53:41.979091 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:41.928104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52333 (local address 127.0.97.62:34335)
0929 20:53:41.946727 (+ 18623us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:41.946732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:41.946750 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:53:41.946801 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:41.946805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:41.946875 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:41.946966 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:41.949606 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.950741 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.957582 (+ 6841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.958240 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.958327 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:41.971141 (+ 12814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:41.971147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:41.971151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:41.971195 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:41.975434 (+ 4239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:41.976211 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:41.976217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:41.976220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:41.976379 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:41.976781 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:41.976786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:41.976789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:41.977074 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:41.977186 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:41.978535 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0929 20:53:41.978743 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18517,"thread_start_us":18452,"threads_started":1}
Sep 29 20:53:41 dist-test-slave-m9g1 krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 20:53:41.980118 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:41.949888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55856 (local address 127.0.97.62:34335)
0929 20:53:41.961818 (+ 11930us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:41.961822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:41.961835 (+ 13us) server_negotiation.cc:407] Connection header received
0929 20:53:41.961872 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:41.961875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:41.961923 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:41.962013 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:41.964053 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.965231 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.967211 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.967876 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.967996 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:41.971497 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:41.971504 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:41.971508 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:41.971546 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:41.976366 (+ 4820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:41.977118 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:41.977123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:41.977126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:41.977256 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:41.977657 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:41.977661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:41.977665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:41.977971 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:41.978180 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:41.979519 (+ 1339us) server_negotiation.cc:299] Negotiation successful
0929 20:53:41.979736 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11838,"thread_start_us":11747,"threads_started":1}
I20260929 20:53:41.980216 7769 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34335
I20260929 20:53:41.980329 7769 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:41.980736 7769 heartbeater.cc:511] Master 127.0.97.62:34335 requested a full tablet report, sending...
I20260929 20:53:41.981665 6516 ts_manager.cc:194] Re-registered known tserver with Master: 3529304287894649b2750ee4aeb0fa05 (127.0.97.3:45645)
I20260929 20:53:41.982679 6516 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52333
I20260929 20:53:41.984211 6516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 20:53:41.993291 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:41.964177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60958 (local address 127.0.97.61:40257)
0929 20:53:41.973311 (+ 9134us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:41.973317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:41.982884 (+ 9567us) server_negotiation.cc:407] Connection header received
0929 20:53:41.985421 (+ 2537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:41.985427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:41.985603 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:41.985709 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:41.986448 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.987488 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.989590 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.990277 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.990365 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:41.991944 (+ 1579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:41.992614 (+ 670us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:41.992716 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:41.992935 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 20:53:41.993018 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9018,"thread_start_us":20905,"threads_started":1}
I20260929 20:53:41.994310 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:41.943221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34053 (local address 127.0.97.60:41713)
0929 20:53:41.946540 (+ 3319us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:41.946546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:41.946564 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:53:41.946622 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:41.946627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:41.946705 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:41.947322 (+ 617us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:41.949597 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.950737 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.971448 (+ 20711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.972101 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.972228 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:41.988271 (+ 16043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:41.988279 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:41.988285 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:41.988330 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:41.991432 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:41.992217 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:41.992223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:41.992227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:41.992363 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:41.992749 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:41.992754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:41.992757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:41.993102 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:41.993204 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:41.993704 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 20:53:41.993920 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3213,"thread_start_us":3132,"threads_started":1}
W20260929 20:53:41.994570 6516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:53:41.995261 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:41.942843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41713 (local address 127.0.97.3:34053)
0929 20:53:41.943324 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:41.943342 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:41.943452 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:41.947147 (+ 3695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:41.947160 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:41.948459 (+ 1299us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:41.949402 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:41.949426 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.962319 (+ 12893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.962324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:41.963732 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:41.963745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.972542 (+ 8797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.972548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:41.972708 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:41.978313 (+ 5605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:41.978333 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:41.983759 (+ 5426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:41.991562 (+ 7803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:41.991568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:41.991572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:41.992080 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:41.992503 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:41.992508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:41.992512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:41.992648 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:41.993293 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:41.993298 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:41.993443 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 20:53:41.994538 (+ 1095us) client_negotiation.cc:240] Negotiation successful
0929 20:53:41.994870 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":116,"thread_start_us":515,"threads_started":1}
I20260929 20:53:41.996050 7768 heartbeater.cc:347] Connected to a master server at 127.0.97.60:41713
I20260929 20:53:41.996176 7768 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:41.996569 7768 heartbeater.cc:511] Master 127.0.97.60:41713 requested a full tablet report, sending...
I20260929 20:53:41.997720 7302 ts_manager.cc:194] Registered new tserver with Master: 3529304287894649b2750ee4aeb0fa05 (127.0.97.3:45645)
I20260929 20:53:42.006230 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:41.951538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49710 (local address 127.0.97.60:41713)
0929 20:53:41.984480 (+ 32942us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:41.984486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:41.984508 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:53:41.984558 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:41.984563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:41.984639 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:41.989079 (+ 4440us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:41.991640 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.992751 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.995308 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:41.995947 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:41.996063 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:41.999403 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:41.999407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:41.999411 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:41.999436 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:42.002429 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:42.003052 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:42.003056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:42.003059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:42.003214 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:42.003654 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:42.003659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:42.003663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:42.004050 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:42.005365 (+ 1315us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:42.005508 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 20:53:42.005738 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32813,"thread_start_us":32705,"threads_started":1}
I20260929 20:53:42.044569 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:42.024648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36383 (local address 127.0.0.1:49808)
0929 20:53:42.035542 (+ 10894us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:42.035566 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:42.035679 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:42.036790 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:42.036795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:42.036823 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:42.037258 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:42.037270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.039490 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.039496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:42.041574 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:42.041590 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.043773 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.043779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:42.043954 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:42.043984 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:53:42.044079 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 20:53:42.044210 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10772,"thread_start_us":13696,"threads_started":1}
I20260929 20:53:42.044658 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:42.024969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49808 (local address 127.0.97.2:36383)
0929 20:53:42.036102 (+ 11133us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:42.036112 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:42.036146 (+ 34us) server_negotiation.cc:407] Connection header received
0929 20:53:42.036257 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:42.036266 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:42.036560 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:42.036772 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:42.037457 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.039326 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.042267 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.043610 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.043731 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:42.043935 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:42.044249 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 20:53:42.044431 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10989,"thread_start_us":10893,"threads_started":1}
I20260929 20:53:42.045691 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:42.024496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.0.97.3:45645)
0929 20:53:42.037364 (+ 12868us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:42.037372 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:42.037403 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:53:42.037497 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:42.037505 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:42.037758 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:42.037965 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:42.038879 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.040400 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.042243 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.043742 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.043859 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:42.044038 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:42.045209 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0929 20:53:42.045382 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12728,"thread_start_us":12579,"threads_started":1}
I20260929 20:53:42.045859 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:42.024242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45645 (local address 127.0.0.1:51748)
0929 20:53:42.033252 (+ 9010us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:42.033275 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:42.033411 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:42.037938 (+ 4527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:42.037943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:42.037967 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:42.038376 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:42.038387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.040515 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.040520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:42.042055 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:42.042069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.044072 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.044077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:42.044355 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:42.044386 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:53:42.044458 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 20:53:42.044588 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8858,"mutex_wait_us":37,"thread_start_us":8712,"threads_started":1}
I20260929 20:53:42.046931 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:42.024996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46367 (local address 127.0.0.1:44444)
0929 20:53:42.038581 (+ 13585us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:42.038606 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:42.038720 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:42.039698 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:42.039706 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:42.039730 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:42.040127 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:42.040137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.041831 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.041837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:42.044279 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:42.044296 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.046241 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.046247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:42.046374 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:42.046395 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 20:53:42.046471 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 20:53:42.046585 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13478,"mutex_wait_us":39,"thread_start_us":10280,"threads_started":1}
I20260929 20:53:42.046941 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:42.025480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44444 (local address 127.0.97.1:46367)
0929 20:53:42.034614 (+ 9134us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:42.034624 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:42.038710 (+ 4086us) server_negotiation.cc:407] Connection header received
0929 20:53:42.038839 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:42.038844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:42.039001 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:42.039723 (+ 722us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:42.040291 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.041711 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.044449 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.045784 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.045905 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:42.046092 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:42.046528 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 20:53:42.046681 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8979,"thread_start_us":8877,"threads_started":1}
I20260929 20:53:42.048568 7562 tablet_service.cc:1467] Processing CreateTablet for tablet 1910995a15744caba6ae9311ac6e4d8f (DEFAULT_TABLE table=test-table [id=81a7ed42a6664b2b81295dd4ade41c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:42.049973 7562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1910995a15744caba6ae9311ac6e4d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:42.050096 7414 tablet_service.cc:1467] Processing CreateTablet for tablet 1910995a15744caba6ae9311ac6e4d8f (DEFAULT_TABLE table=test-table [id=81a7ed42a6664b2b81295dd4ade41c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:42.050081 7703 tablet_service.cc:1467] Processing CreateTablet for tablet 1910995a15744caba6ae9311ac6e4d8f (DEFAULT_TABLE table=test-table [id=81a7ed42a6664b2b81295dd4ade41c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:42.051195 7414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1910995a15744caba6ae9311ac6e4d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:42.051373 7703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1910995a15744caba6ae9311ac6e4d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:42.062894 7801 tablet_bootstrap.cc:492] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a: Bootstrap starting.
I20260929 20:53:42.065102 7802 tablet_bootstrap.cc:492] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91: Bootstrap starting.
I20260929 20:53:42.066916 7803 tablet_bootstrap.cc:492] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05: Bootstrap starting.
I20260929 20:53:42.068152 7801 tablet_bootstrap.cc:654] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:42.068177 7802 tablet_bootstrap.cc:654] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:42.069518 7802 log.cc:824] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:42.069737 7803 tablet_bootstrap.cc:654] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:42.069959 7801 log.cc:824] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:42.070833 7803 log.cc:824] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:42.072458 7802 tablet_bootstrap.cc:492] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91: No bootstrap required, opened a new log
I20260929 20:53:42.072600 7801 tablet_bootstrap.cc:492] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a: No bootstrap required, opened a new log
I20260929 20:53:42.072697 7802 ts_tablet_manager.cc:1397] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 20:53:42.072880 7801 ts_tablet_manager.cc:1397] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 20:53:42.073536 7803 tablet_bootstrap.cc:492] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05: No bootstrap required, opened a new log
I20260929 20:53:42.073764 7803 ts_tablet_manager.cc:1397] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 20:53:42.080363 7802 raft_consensus.cc:348] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } } peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } }
I20260929 20:53:42.080689 7802 raft_consensus.cc:374] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:42.080772 7802 raft_consensus.cc:729] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3bebe27bd744c1bcd36918f642ca91, State: Initialized, Role: FOLLOWER
I20260929 20:53:42.081317 7802 consensus_queue.cc:260] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } } peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } }
I20260929 20:53:42.082742 7801 raft_consensus.cc:348] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } } peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } }
I20260929 20:53:42.083140 7801 raft_consensus.cc:374] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:42.083222 7801 raft_consensus.cc:729] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e816c455c3944cac8973eaac1bf8580a, State: Initialized, Role: FOLLOWER
I20260929 20:53:42.083384 7802 ts_tablet_manager.cc:1428] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91: Time spent starting tablet: real 0.011s user 0.004s sys 0.007s
I20260929 20:53:42.083652 7628 heartbeater.cc:503] Master 127.0.97.62:34335 was elected leader, sending a full tablet report...
I20260929 20:53:42.083870 7803 raft_consensus.cc:348] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } } peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } }
I20260929 20:53:42.084239 7803 raft_consensus.cc:374] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:42.084333 7803 raft_consensus.cc:729] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3529304287894649b2750ee4aeb0fa05, State: Initialized, Role: FOLLOWER
I20260929 20:53:42.084762 7801 consensus_queue.cc:260] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } } peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } }
I20260929 20:53:42.084915 7803 consensus_queue.cc:260] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } } peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } }
I20260929 20:53:42.090456 7801 ts_tablet_manager.cc:1428] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a: Time spent starting tablet: real 0.017s user 0.008s sys 0.010s
I20260929 20:53:42.092201 7769 heartbeater.cc:503] Master 127.0.97.62:34335 was elected leader, sending a full tablet report...
I20260929 20:53:42.095130 7803 ts_tablet_manager.cc:1428] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05: Time spent starting tablet: real 0.021s user 0.011s sys 0.008s
W20260929 20:53:42.112498 7630 tablet.cc:2369] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:53:42.182952 7771 tablet.cc:2369] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:53:42.244086 7807 raft_consensus.cc:482] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:53:42.244772 7807 raft_consensus.cc:504] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } } peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } }
I20260929 20:53:42.246454 7807 leader_election.cc:288] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3529304287894649b2750ee4aeb0fa05 (127.0.97.3:45645), e816c455c3944cac8973eaac1bf8580a (127.0.97.1:46367)
I20260929 20:53:42.252786 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:42.246812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48755 (local address 127.0.97.1:46367)
0929 20:53:42.247119 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:42.247125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:42.247680 (+ 555us) server_negotiation.cc:407] Connection header received
0929 20:53:42.247885 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:42.247891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:42.247977 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:42.248063 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:42.248646 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.249354 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.251196 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.251945 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.252055 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:42.252145 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:42.252426 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 20:53:42.252549 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260929 20:53:42.252789 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:42.246793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33801 (local address 127.0.97.3:45645)
0929 20:53:42.246983 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:42.246989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:42.247971 (+ 982us) server_negotiation.cc:407] Connection header received
0929 20:53:42.248136 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:42.248142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:42.248214 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:42.248302 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:42.248754 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.249483 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.251210 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.251951 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.252056 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:42.252154 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:42.252438 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 20:53:42.252554 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:53:42.252918 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:42.246823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45645 (local address 127.0.97.2:33801)
0929 20:53:42.247868 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:42.247889 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:42.248007 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:42.248324 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:42.248327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:42.248348 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:42.248613 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:42.248623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.249608 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.249613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:42.251059 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:42.251070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.252181 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.252187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:42.252306 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:42.252327 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 20:53:42.252402 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 20:53:42.252546 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":42,"thread_start_us":774,"threads_started":1}
I20260929 20:53:42.253228 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:42.246691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46367 (local address 127.0.97.2:48755)
0929 20:53:42.247546 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:42.247567 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:42.247733 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:42.248081 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:42.248085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:42.248111 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:42.248485 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:42.248498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.249502 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.249506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:42.251047 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:42.251058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.252146 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.252153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:42.252272 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:42.252300 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 20:53:42.252375 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 20:53:42.252511 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":51,"thread_start_us":471,"threads_started":1}
I20260929 20:53:42.253806 7723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1910995a15744caba6ae9311ac6e4d8f" candidate_uuid: "9f3bebe27bd744c1bcd36918f642ca91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3529304287894649b2750ee4aeb0fa05" is_pre_election: true
I20260929 20:53:42.253995 7434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1910995a15744caba6ae9311ac6e4d8f" candidate_uuid: "9f3bebe27bd744c1bcd36918f642ca91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e816c455c3944cac8973eaac1bf8580a" is_pre_election: true
I20260929 20:53:42.254305 7723 raft_consensus.cc:2454] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f3bebe27bd744c1bcd36918f642ca91 in term 0.
I20260929 20:53:42.254447 7434 raft_consensus.cc:2454] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f3bebe27bd744c1bcd36918f642ca91 in term 0.
I20260929 20:53:42.254966 7516 leader_election.cc:302] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3529304287894649b2750ee4aeb0fa05, 9f3bebe27bd744c1bcd36918f642ca91; no voters:
I20260929 20:53:42.255384 7807 raft_consensus.cc:2781] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:53:42.255944 7807 raft_consensus.cc:482] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:53:42.256038 7807 raft_consensus.cc:3037] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:42.258252 7807 raft_consensus.cc:504] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } } peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } }
I20260929 20:53:42.258785 7807 leader_election.cc:288] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [CANDIDATE]: Term 1 election: Requested vote from peers 3529304287894649b2750ee4aeb0fa05 (127.0.97.3:45645), e816c455c3944cac8973eaac1bf8580a (127.0.97.1:46367)
I20260929 20:53:42.259217 7434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1910995a15744caba6ae9311ac6e4d8f" candidate_uuid: "9f3bebe27bd744c1bcd36918f642ca91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e816c455c3944cac8973eaac1bf8580a"
I20260929 20:53:42.259244 7723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1910995a15744caba6ae9311ac6e4d8f" candidate_uuid: "9f3bebe27bd744c1bcd36918f642ca91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3529304287894649b2750ee4aeb0fa05"
I20260929 20:53:42.259380 7434 raft_consensus.cc:3037] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:42.259411 7723 raft_consensus.cc:3037] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:42.261468 7434 raft_consensus.cc:2454] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f3bebe27bd744c1bcd36918f642ca91 in term 1.
I20260929 20:53:42.261560 7723 raft_consensus.cc:2454] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f3bebe27bd744c1bcd36918f642ca91 in term 1.
I20260929 20:53:42.261830 7517 leader_election.cc:302] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f3bebe27bd744c1bcd36918f642ca91, e816c455c3944cac8973eaac1bf8580a; no voters:
I20260929 20:53:42.262108 7807 raft_consensus.cc:2781] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:42.262540 7807 raft_consensus.cc:686] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [term 1 LEADER]: Becoming Leader. State: Replica: 9f3bebe27bd744c1bcd36918f642ca91, State: Running, Role: LEADER
I20260929 20:53:42.262967 7807 consensus_queue.cc:237] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } } peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } }
I20260929 20:53:42.267608 6515 catalog_manager.cc:5495] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f3bebe27bd744c1bcd36918f642ca91 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "9f3bebe27bd744c1bcd36918f642ca91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } health_report { overall_health: UNKNOWN } } }
W20260929 20:53:42.275892 7485 tablet.cc:2369] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:53:42.339314 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:42.334354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.0.97.2:36383)
0929 20:53:42.334510 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:42.334514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:42.334530 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:53:42.334615 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:42.334618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:42.334688 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:42.334803 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:42.335170 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.335942 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.337247 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.337631 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.337758 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:42.338219 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:42.338791 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:42.338899 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:42.338998 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 20:53:42.339063 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 20:53:42.347234 7723 raft_consensus.cc:1261] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05 [term 1 FOLLOWER]: Refusing update from remote peer 9f3bebe27bd744c1bcd36918f642ca91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:42.347234 7434 raft_consensus.cc:1261] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a [term 1 FOLLOWER]: Refusing update from remote peer 9f3bebe27bd744c1bcd36918f642ca91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:42.348062 7812 consensus_queue.cc:1035] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:42.348392 7807 consensus_queue.cc:1035] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:42.382561 7815 mvcc.cc:205] Tried to move back new op lower bound from 7334769550724939776 to 7334769550393073664. Current Snapshot: MvccSnapshot[applied={T|T < 7334769550724939776}]
I20260929 20:53:42.402333 6516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:42.402559 6516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:42.408071 6516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 347ec01fc4094722a63b6055158d723c: Sending DeleteTablet for 3 replicas of tablet 1910995a15744caba6ae9311ac6e4d8f
I20260929 20:53:42.409008 7703 tablet_service.cc:1514] Processing DeleteTablet for tablet 1910995a15744caba6ae9311ac6e4d8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:42 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51748
I20260929 20:53:42.410148 7414 tablet_service.cc:1514] Processing DeleteTablet for tablet 1910995a15744caba6ae9311ac6e4d8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:42 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44444
I20260929 20:53:42.410959 7562 tablet_service.cc:1514] Processing DeleteTablet for tablet 1910995a15744caba6ae9311ac6e4d8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:42 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49808
I20260929 20:53:42.414356 7825 tablet_replica.cc:330] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a: stopping tablet replica
I20260929 20:53:42.414901 7825 raft_consensus.cc:2229] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:42.415432 7825 raft_consensus.cc:2258] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:42.417830 7825 ts_tablet_manager.cc:1905] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:42.421840 7826 tablet_replica.cc:330] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91: stopping tablet replica
I20260929 20:53:42.422312 7826 raft_consensus.cc:2229] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:53:42.422948 7826 raft_consensus.cc:2258] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:42.423004 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 6484
I20260929 20:53:42.424484 6546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:42.424821 7826 ts_tablet_manager.cc:1905] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:42.430172 7824 tablet_replica.cc:330] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05: stopping tablet replica
I20260929 20:53:42.430698 7824 raft_consensus.cc:2229] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:42.431170 7824 raft_consensus.cc:2258] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:42.431627 7826 ts_tablet_manager.cc:1918] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:42.431798 7826 log.cc:1196] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/1910995a15744caba6ae9311ac6e4d8f
I20260929 20:53:42.432330 7826 ts_tablet_manager.cc:1939] T 1910995a15744caba6ae9311ac6e4d8f P 9f3bebe27bd744c1bcd36918f642ca91: Deleting consensus metadata
I20260929 20:53:42.433910 7825 ts_tablet_manager.cc:1918] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:42.434072 7825 log.cc:1196] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/1910995a15744caba6ae9311ac6e4d8f
I20260929 20:53:42.434590 7825 ts_tablet_manager.cc:1939] T 1910995a15744caba6ae9311ac6e4d8f P e816c455c3944cac8973eaac1bf8580a: Deleting consensus metadata
I20260929 20:53:42.442209 7824 ts_tablet_manager.cc:1905] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:42.442651 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:42.411780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.0.97.61:40257)
0929 20:53:42.411945 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:42.411952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:42.421818 (+ 9866us) server_negotiation.cc:407] Connection header received
0929 20:53:42.424208 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:42.424215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:42.424329 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:42.424454 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:42.427324 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.428528 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.430579 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.431165 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.431274 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:42.435669 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:42.435676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:42.435682 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:42.435724 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:42.439078 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:42.439878 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:42.439885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:42.439890 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:42.440036 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:42.440424 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:42.440429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:42.440432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:42.440777 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:42.440992 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:42.442076 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0929 20:53:42.442307 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:53:42.456442 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:42.421919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.0.97.60:41713)
0929 20:53:42.422065 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:42.422071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:42.441853 (+ 19782us) server_negotiation.cc:407] Connection header received
0929 20:53:42.441977 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:42.441982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:42.442076 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:42.442164 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:42.443915 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.444625 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.445838 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.446296 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.446412 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:42.449856 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:42.449863 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:42.449868 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:42.449905 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:42.453255 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:42.454027 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:42.454033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:42.454036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:42.454248 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:42.454784 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:42.454789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:42.454792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:42.455102 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:42.455311 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:42.455898 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 20:53:42.456114 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 20:53:42.457132 7824 ts_tablet_manager.cc:1918] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:42.457345 7824 log.cc:1196] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/1910995a15744caba6ae9311ac6e4d8f
I20260929 20:53:42.457834 7824 ts_tablet_manager.cc:1939] T 1910995a15744caba6ae9311ac6e4d8f P 3529304287894649b2750ee4aeb0fa05: Deleting consensus metadata
I20260929 20:53:42.559962 6501 catalog_manager.cc:4841] TS 3529304287894649b2750ee4aeb0fa05 (127.0.97.3:45645): tablet 1910995a15744caba6ae9311ac6e4d8f (table test-table [id=81a7ed42a6664b2b81295dd4ade41c49]) successfully deleted
I20260929 20:53:42.560681 6502 catalog_manager.cc:4841] TS e816c455c3944cac8973eaac1bf8580a (127.0.97.1:46367): tablet 1910995a15744caba6ae9311ac6e4d8f (table test-table [id=81a7ed42a6664b2b81295dd4ade41c49]) successfully deleted
W20260929 20:53:42.560948 6502 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.1:46367, user_credentials={real_user=slave}} blocked reactor thread for 124789us
I20260929 20:53:42.559999 6503 catalog_manager.cc:4841] TS 9f3bebe27bd744c1bcd36918f642ca91 (127.0.97.2:36383): tablet 1910995a15744caba6ae9311ac6e4d8f (table test-table [id=81a7ed42a6664b2b81295dd4ade41c49]) successfully deleted
W20260929 20:53:42.561956 6503 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.2:36383, user_credentials={real_user=slave}} blocked reactor thread for 127986us
I20260929 20:53:42.562240 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 6484
I20260929 20:53:42.610860 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:42.590976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49730 (local address 127.0.97.60:41713)
0929 20:53:42.591203 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:42.591210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:42.594124 (+ 2914us) server_negotiation.cc:407] Connection header received
0929 20:53:42.594640 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:42.594646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:42.594736 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:42.594806 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:42.597505 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.598638 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.601043 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.601663 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.601756 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:42.605538 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:42.605546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:42.605550 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:42.605590 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:42.608246 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:42.608840 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:42.608844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:42.608846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:42.608978 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:42.609313 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:42.609318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:42.609322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:42.609676 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:42.609906 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:42.610336 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 20:53:42.610514 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:53:42.612298 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:42.590993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60972 (local address 127.0.97.61:40257)
0929 20:53:42.591339 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:42.591345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:42.597328 (+ 5983us) server_negotiation.cc:407] Connection header received
0929 20:53:42.597469 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:42.597475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:42.597560 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:42.597658 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:42.600630 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.601551 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.603058 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:42.603608 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:42.603743 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:42.606229 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:42.606237 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:42.606241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:42.606280 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:42.609505 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:42.610236 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:42.610241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:42.610245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:42.610375 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:42.610815 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:42.610820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:42.610824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:42.611212 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:42.611476 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:42.611752 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 20:53:42.611989 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
I20260929 20:53:43.394232 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:43.393510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.2:57145)
0929 20:53:43.393930 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:43.394045 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":211,"threads_started":1}
W20260929 20:53:43.394599 7628 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34335 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
I20260929 20:53:43.396250 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:43.395661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.1:56783)
0929 20:53:43.396029 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:43.396122 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":179,"threads_started":1}
W20260929 20:53:43.396598 7483 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34335 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
I20260929 20:53:43.405268 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:43.404484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.3:43393)
0929 20:53:43.404930 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:43.405073 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":236,"threads_started":1}
W20260929 20:53:43.405578 7769 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34335 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
I20260929 20:53:43.909936 7839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 347ec01fc4094722a63b6055158d723c)
I20260929 20:53:43.910202 7839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: false } }
I20260929 20:53:43.910859 7840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 347ec01fc4094722a63b6055158d723c)
I20260929 20:53:43.911015 7840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: false } }
I20260929 20:53:43.911830 7839 leader_election.cc:288] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 347ec01fc4094722a63b6055158d723c (127.0.97.62:34335), 42034594181341f4b03369c668783273 (127.0.97.60:41713)
I20260929 20:53:43.918686 7840 leader_election.cc:288] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 347ec01fc4094722a63b6055158d723c (127.0.97.62:34335), 60d848cea40b4d6e9c79fdda2fa9c401 (127.0.97.61:40257)
I20260929 20:53:43.925825 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:43.912025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.0.1:56256)
0929 20:53:43.925348 (+ 13323us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:43.925533 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13214,"thread_start_us":9475,"threads_started":1}
W20260929 20:53:43.927403 7288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
I20260929 20:53:43.932788 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:43.911976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.0.1:56250)
0929 20:53:43.932257 (+ 20281us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:43.932449 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20148,"threads_started":1}
I20260929 20:53:43.932814 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:43.911997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40257 (local address 127.0.0.1:60984)
0929 20:53:43.921776 (+ 9779us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:43.921798 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:43.921944 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:43.926076 (+ 4132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:43.926083 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:43.926110 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:43.926422 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:43.926434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:43.928122 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:43.928127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:43.930267 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:43.930282 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:43.932052 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:43.932058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:43.932210 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:43.932242 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:53:43.932322 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 20:53:43.932460 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9648,"thread_start_us":13110,"threads_started":1}
I20260929 20:53:43.933498 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:43.912135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.0.97.61:40257)
0929 20:53:43.925626 (+ 13491us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:43.925631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:43.925650 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:53:43.925717 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:43.925722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:43.925788 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:43.925880 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:43.926617 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:43.927938 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:43.930454 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:43.931717 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:43.931833 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:43.931951 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:43.933038 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0929 20:53:43.933169 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13378,"thread_start_us":13205,"threads_started":1}
I20260929 20:53:43.934134 7124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42034594181341f4b03369c668783273" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" is_pre_election: true
I20260929 20:53:43.934459 7124 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42034594181341f4b03369c668783273 in term 1.
I20260929 20:53:43.934902 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:43.934468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.0.1:56274)
0929 20:53:43.934651 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:43.934743 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 20:53:43.935063 7289 leader_election.cc:302] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42034594181341f4b03369c668783273, 60d848cea40b4d6e9c79fdda2fa9c401; no voters:
I20260929 20:53:43.935390 7840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 20:53:43.935505 7840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 347ec01fc4094722a63b6055158d723c)
I20260929 20:53:43.935617 7840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 1 FOLLOWER]: Advancing to term 2
W20260929 20:53:43.935528 7101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
W20260929 20:53:43.935760 7288 leader_election.cc:334] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 347ec01fc4094722a63b6055158d723c (127.0.97.62:34335): Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
I20260929 20:53:43.937359 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:43.936846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.0.1:56284)
0929 20:53:43.937005 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:43.937162 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 20:53:43.937558 7101 leader_election.cc:334] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 347ec01fc4094722a63b6055158d723c (127.0.97.62:34335): Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
I20260929 20:53:43.938647 7840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: false } }
I20260929 20:53:43.939313 7840 leader_election.cc:288] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [CANDIDATE]: Term 2 election: Requested vote from peers 347ec01fc4094722a63b6055158d723c (127.0.97.62:34335), 60d848cea40b4d6e9c79fdda2fa9c401 (127.0.97.61:40257)
I20260929 20:53:43.939698 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:43.939351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.0.1:56296)
0929 20:53:43.939521 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:43.939585 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 20:53:43.939693 7124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42034594181341f4b03369c668783273" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "60d848cea40b4d6e9c79fdda2fa9c401"
I20260929 20:53:43.939874 7124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:53:43.940622 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:43.940262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.0.1:56300)
0929 20:53:43.940415 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:43.940494 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 20:53:43.940833 7288 leader_election.cc:334] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 347ec01fc4094722a63b6055158d723c (127.0.97.62:34335): Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
I20260929 20:53:43.941669 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:43.914240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41713 (local address 127.0.0.1:49738)
0929 20:53:43.933335 (+ 19095us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:43.933355 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:43.933484 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:43.935064 (+ 1580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:43.935069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:43.935093 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:43.935449 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:43.935458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:43.937099 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:43.937105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:43.939059 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:43.939072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:43.940984 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:43.940989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:43.941147 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:43.941169 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 20:53:43.941247 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 20:53:43.941396 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18969,"thread_start_us":17369,"threads_started":1}
I20260929 20:53:43.941684 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:43.916577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49738 (local address 127.0.97.60:41713)
0929 20:53:43.931889 (+ 15312us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:43.931895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:43.934398 (+ 2503us) server_negotiation.cc:407] Connection header received
0929 20:53:43.934462 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:43.934466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:43.934528 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:43.934725 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:43.935621 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:43.936966 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:43.939218 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:43.940807 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:43.940936 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:43.941030 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:43.941277 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 20:53:43.941393 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15164,"thread_start_us":16501,"threads_started":1}
I20260929 20:53:43.942333 7124 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42034594181341f4b03369c668783273 in term 2.
I20260929 20:53:43.942250 7312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "42034594181341f4b03369c668783273" is_pre_election: true
I20260929 20:53:43.942613 7312 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d848cea40b4d6e9c79fdda2fa9c401 in current term 2: Already voted for candidate 42034594181341f4b03369c668783273 in this term.
I20260929 20:53:43.942658 7289 leader_election.cc:302] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42034594181341f4b03369c668783273, 60d848cea40b4d6e9c79fdda2fa9c401; no voters: 347ec01fc4094722a63b6055158d723c
I20260929 20:53:43.942971 7840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:53:43.943131 7099 leader_election.cc:302] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d848cea40b4d6e9c79fdda2fa9c401; no voters: 347ec01fc4094722a63b6055158d723c, 42034594181341f4b03369c668783273
I20260929 20:53:43.943387 7840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [term 2 LEADER]: Becoming Leader. State: Replica: 42034594181341f4b03369c668783273, State: Running, Role: LEADER
I20260929 20:53:43.943493 7839 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:53:43.943787 7840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: false } }
I20260929 20:53:43.945420 7849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42034594181341f4b03369c668783273. Latest consensus state: current_term: 2 leader_uuid: "42034594181341f4b03369c668783273" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: false } } }
I20260929 20:53:43.945770 7849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:43.946341 7851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:53:43.949579 7851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:53:43.950362 7851 catalog_manager.cc:1259] Loaded cluster ID: 4418e57d94d541228f2db15ec1ae6260
I20260929 20:53:43.950429 7851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:53:43.951467 7851 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:53:43.952206 7851 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: Loaded TSK: 0
I20260929 20:53:43.952968 7851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:53:43.992111 7302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:53:43.999053 7302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:53:44.002946 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.002543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.0.1:56318)
0929 20:53:44.002714 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.002797 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 20:53:44.003047 7124 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [term 2 FOLLOWER]: Refusing update from remote peer 42034594181341f4b03369c668783273: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 20:53:44.003805 7849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 20:53:44.003964 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.003571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.0.1:56330)
0929 20:53:44.003741 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.003812 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 20:53:44.004113 7288 consensus_peers.cc:487] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 -> Peer 347ec01fc4094722a63b6055158d723c (127.0.97.62:34335): Couldn't send request to peer 347ec01fc4094722a63b6055158d723c. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 20:53:44.006626 7839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42034594181341f4b03369c668783273. Latest consensus state: current_term: 2 leader_uuid: "42034594181341f4b03369c668783273" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: false } } }
I20260929 20:53:44.006898 7839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:53:44.007977 7840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "42034594181341f4b03369c668783273" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: false } } }
I20260929 20:53:44.008318 7840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:44.008637 7839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "42034594181341f4b03369c668783273" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "347ec01fc4094722a63b6055158d723c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34335 } } peers { permanent_uuid: "60d848cea40b4d6e9c79fdda2fa9c401" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40257 } attrs { promote: false } } peers { permanent_uuid: "42034594181341f4b03369c668783273" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41713 } attrs { promote: false } } }
I20260929 20:53:44.008876 7839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d848cea40b4d6e9c79fdda2fa9c401 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:53:44.023554 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.015947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36383 (local address 127.0.0.1:49818)
0929 20:53:44.016063 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.016079 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:44.016158 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:44.017003 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:44.017008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:44.017037 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:44.017245 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:44.017255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.018871 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.018878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:44.020957 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:44.020973 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.022707 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.022714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:44.022940 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:44.022982 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 20:53:44.023049 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 20:53:44.023194 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 20:53:44.024845 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.016247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.0.97.2:36383)
0929 20:53:44.016658 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:44.016664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:44.016681 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:53:44.016747 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:44.016753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:44.016822 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:44.017372 (+ 550us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:44.017533 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.018696 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.021239 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.022531 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.024200 (+ 1669us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:44.024319 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:44.024371 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 20:53:44.024520 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":750,"threads_started":1}
I20260929 20:53:44.025516 7562 tablet_service.cc:1467] Processing CreateTablet for tablet 39c37e992fd04bacbdf820820b74fabd (DEFAULT_TABLE table=test-table [id=bc22e3bc235448e98f2a405ca25f5135]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:44.025885 7562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c37e992fd04bacbdf820820b74fabd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:44.034112 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.016593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46367 (local address 127.0.0.1:44446)
0929 20:53:44.016710 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.016726 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:44.016857 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:44.027653 (+ 10796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:44.027658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:44.027687 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:44.028007 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:44.028017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.029484 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.029490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:44.031449 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:44.031464 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.033362 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.033369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:44.033550 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:44.033585 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 20:53:44.033671 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 20:53:44.033819 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 20:53:44.034890 7862 tablet_bootstrap.cc:492] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91: Bootstrap starting.
I20260929 20:53:44.035269 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.016881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44446 (local address 127.0.97.1:46367)
0929 20:53:44.027226 (+ 10345us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:44.027234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:44.027256 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:53:44.027338 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:44.027343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:44.027418 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:44.027531 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:44.028154 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.029331 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.031631 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.032802 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.032934 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:44.033036 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:44.033719 (+ 683us) server_negotiation.cc:299] Negotiation successful
0929 20:53:44.034956 (+ 1237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10212,"thread_start_us":10757,"threads_started":1}
I20260929 20:53:44.035915 7414 tablet_service.cc:1467] Processing CreateTablet for tablet 39c37e992fd04bacbdf820820b74fabd (DEFAULT_TABLE table=test-table [id=bc22e3bc235448e98f2a405ca25f5135]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:44.036345 7414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c37e992fd04bacbdf820820b74fabd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:44.036445 7862 tablet_bootstrap.cc:654] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:44.038244 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.016651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45645 (local address 127.0.0.1:51752)
0929 20:53:44.023805 (+ 7154us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.023824 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:44.023937 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:44.032930 (+ 8993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:44.032936 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:44.032966 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:44.033143 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:44.033152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.034219 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.034224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:44.035887 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:44.035900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.037480 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.037486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:44.037704 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:44.037736 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:53:44.037822 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 20:53:44.037954 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7056,"thread_start_us":13562,"threads_started":1}
I20260929 20:53:44.038425 7862 tablet_bootstrap.cc:492] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91: No bootstrap required, opened a new log
I20260929 20:53:44.038282 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.017072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.0.97.3:45645)
0929 20:53:44.032640 (+ 15568us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:44.032647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:44.032665 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:53:44.032737 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:44.032742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:44.032827 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:44.032913 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:44.033266 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.034050 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.036225 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.037310 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.037423 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:44.037518 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:44.037855 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 20:53:44.037982 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15444,"thread_start_us":15953,"threads_started":1}
I20260929 20:53:44.038566 7862 ts_tablet_manager.cc:1397] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260929 20:53:44.039294 7703 tablet_service.cc:1467] Processing CreateTablet for tablet 39c37e992fd04bacbdf820820b74fabd (DEFAULT_TABLE table=test-table [id=bc22e3bc235448e98f2a405ca25f5135]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:44.039690 7703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c37e992fd04bacbdf820820b74fabd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:44.042881 7864 tablet_bootstrap.cc:492] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a: Bootstrap starting.
I20260929 20:53:44.042840 7862 raft_consensus.cc:348] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } } peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } }
I20260929 20:53:44.043134 7862 raft_consensus.cc:374] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:44.043234 7862 raft_consensus.cc:729] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3bebe27bd744c1bcd36918f642ca91, State: Initialized, Role: FOLLOWER
I20260929 20:53:44.043485 7862 consensus_queue.cc:260] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } } peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } }
I20260929 20:53:44.045306 7864 tablet_bootstrap.cc:654] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:44.045542 7866 tablet_bootstrap.cc:492] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05: Bootstrap starting.
I20260929 20:53:44.046100 7862 ts_tablet_manager.cc:1428] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91: Time spent starting tablet: real 0.007s user 0.000s sys 0.004s
I20260929 20:53:44.047880 7866 tablet_bootstrap.cc:654] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:44.049925 7627 heartbeater.cc:503] Master 127.0.97.60:41713 was elected leader, sending a full tablet report...
I20260929 20:53:44.051841 7864 tablet_bootstrap.cc:492] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a: No bootstrap required, opened a new log
I20260929 20:53:44.052014 7864 ts_tablet_manager.cc:1397] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20260929 20:53:44.052189 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.047080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.2:59163)
0929 20:53:44.051848 (+ 4768us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.051955 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4657,"thread_start_us":4504,"threads_started":1}
I20260929 20:53:44.052671 7866 tablet_bootstrap.cc:492] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05: No bootstrap required, opened a new log
I20260929 20:53:44.052812 7866 ts_tablet_manager.cc:1397] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 20:53:44.052791 7864 raft_consensus.cc:348] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } } peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } }
I20260929 20:53:44.053030 7864 raft_consensus.cc:374] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:44.053120 7864 raft_consensus.cc:729] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e816c455c3944cac8973eaac1bf8580a, State: Initialized, Role: FOLLOWER
I20260929 20:53:44.053416 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.052681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.2:40269)
0929 20:53:44.053224 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.053304 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":294,"threads_started":1}
I20260929 20:53:44.053380 7864 consensus_queue.cc:260] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } } peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } }
I20260929 20:53:44.053563 7866 raft_consensus.cc:348] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } } peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } }
I20260929 20:53:44.053819 7866 raft_consensus.cc:374] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:44.053925 7866 raft_consensus.cc:729] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3529304287894649b2750ee4aeb0fa05, State: Initialized, Role: FOLLOWER
I20260929 20:53:44.054176 7866 consensus_queue.cc:260] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } } peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } }
I20260929 20:53:44.060998 7864 ts_tablet_manager.cc:1428] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a: Time spent starting tablet: real 0.009s user 0.000s sys 0.003s
I20260929 20:53:44.061308 7482 heartbeater.cc:503] Master 127.0.97.60:41713 was elected leader, sending a full tablet report...
I20260929 20:53:44.063046 7866 ts_tablet_manager.cc:1428] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05: Time spent starting tablet: real 0.010s user 0.000s sys 0.011s
I20260929 20:53:44.063110 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.054975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.1:48567)
0929 20:53:44.062681 (+ 7706us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.062845 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7571,"thread_start_us":7402,"threads_started":1}
I20260929 20:53:44.064311 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.063416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.3:55963)
0929 20:53:44.063985 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.064093 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"threads_started":1}
I20260929 20:53:44.064828 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.064365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.1:58619)
0929 20:53:44.064621 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.064697 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260929 20:53:44.065346 7768 heartbeater.cc:503] Master 127.0.97.60:41713 was elected leader, sending a full tablet report...
I20260929 20:53:44.065606 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.065251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.3:46569)
0929 20:53:44.065388 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.065473 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 20:53:44.251624 7865 raft_consensus.cc:482] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:53:44.251829 7865 raft_consensus.cc:504] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } } peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } }
I20260929 20:53:44.252452 7865 leader_election.cc:288] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e816c455c3944cac8973eaac1bf8580a (127.0.97.1:46367), 3529304287894649b2750ee4aeb0fa05 (127.0.97.3:45645)
I20260929 20:53:44.253007 7434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c37e992fd04bacbdf820820b74fabd" candidate_uuid: "9f3bebe27bd744c1bcd36918f642ca91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e816c455c3944cac8973eaac1bf8580a" is_pre_election: true
I20260929 20:53:44.253014 7723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c37e992fd04bacbdf820820b74fabd" candidate_uuid: "9f3bebe27bd744c1bcd36918f642ca91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3529304287894649b2750ee4aeb0fa05" is_pre_election: true
I20260929 20:53:44.253216 7434 raft_consensus.cc:2454] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f3bebe27bd744c1bcd36918f642ca91 in term 0.
I20260929 20:53:44.253219 7723 raft_consensus.cc:2454] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f3bebe27bd744c1bcd36918f642ca91 in term 0.
I20260929 20:53:44.253625 7517 leader_election.cc:302] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f3bebe27bd744c1bcd36918f642ca91, e816c455c3944cac8973eaac1bf8580a; no voters:
I20260929 20:53:44.253867 7865 raft_consensus.cc:2781] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:53:44.253989 7865 raft_consensus.cc:482] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:53:44.254077 7865 raft_consensus.cc:3037] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:44.256022 7865 raft_consensus.cc:504] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } } peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } }
I20260929 20:53:44.256484 7865 leader_election.cc:288] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [CANDIDATE]: Term 1 election: Requested vote from peers e816c455c3944cac8973eaac1bf8580a (127.0.97.1:46367), 3529304287894649b2750ee4aeb0fa05 (127.0.97.3:45645)
I20260929 20:53:44.256878 7434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c37e992fd04bacbdf820820b74fabd" candidate_uuid: "9f3bebe27bd744c1bcd36918f642ca91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e816c455c3944cac8973eaac1bf8580a"
I20260929 20:53:44.257023 7434 raft_consensus.cc:3037] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:44.256971 7723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c37e992fd04bacbdf820820b74fabd" candidate_uuid: "9f3bebe27bd744c1bcd36918f642ca91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3529304287894649b2750ee4aeb0fa05"
I20260929 20:53:44.257150 7723 raft_consensus.cc:3037] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:44.259086 7723 raft_consensus.cc:2454] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f3bebe27bd744c1bcd36918f642ca91 in term 1.
I20260929 20:53:44.259096 7434 raft_consensus.cc:2454] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f3bebe27bd744c1bcd36918f642ca91 in term 1.
I20260929 20:53:44.259438 7517 leader_election.cc:302] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f3bebe27bd744c1bcd36918f642ca91, e816c455c3944cac8973eaac1bf8580a; no voters:
I20260929 20:53:44.259676 7865 raft_consensus.cc:2781] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:44.260032 7865 raft_consensus.cc:686] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [term 1 LEADER]: Becoming Leader. State: Replica: 9f3bebe27bd744c1bcd36918f642ca91, State: Running, Role: LEADER
I20260929 20:53:44.260280 7865 consensus_queue.cc:237] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } } peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } }
I20260929 20:53:44.260962 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.260566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.2:33507)
0929 20:53:44.260738 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.260816 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 20:53:44.263124 7302 catalog_manager.cc:5495] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f3bebe27bd744c1bcd36918f642ca91 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "9f3bebe27bd744c1bcd36918f642ca91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3bebe27bd744c1bcd36918f642ca91" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 } health_report { overall_health: UNKNOWN } } }
I20260929 20:53:44.329980 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.325677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49820 (local address 127.0.97.2:36383)
0929 20:53:44.325868 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:44.325873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:44.325891 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:53:44.325990 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:44.325994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:44.326059 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:44.326164 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:44.326518 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.327193 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.328423 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.328870 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.328978 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:44.329342 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:44.329450 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:44.329520 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:44.329668 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 20:53:44.329740 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:53:44.332783 7434 raft_consensus.cc:1261] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a [term 1 FOLLOWER]: Refusing update from remote peer 9f3bebe27bd744c1bcd36918f642ca91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:44.332758 7723 raft_consensus.cc:1261] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05 [term 1 FOLLOWER]: Refusing update from remote peer 9f3bebe27bd744c1bcd36918f642ca91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:44.333395 7875 consensus_queue.cc:1035] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3529304287894649b2750ee4aeb0fa05" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:44.333710 7865 consensus_queue.cc:1035] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e816c455c3944cac8973eaac1bf8580a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:44.336112 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.335604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.3:57125)
0929 20:53:44.335838 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.335928 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260929 20:53:44.336118 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.335488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.1:53813)
0929 20:53:44.335895 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.335987 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308}
I20260929 20:53:44.339859 7880 mvcc.cc:205] Tried to move back new op lower bound from 7334769558862209024 to 7334769558572679168. Current Snapshot: MvccSnapshot[applied={T|T < 7334769558862209024}]
I20260929 20:53:44.341068 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.340593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.1:45299)
0929 20:53:44.340846 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.340926 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260929 20:53:44.341303 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.340924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.2:44333)
0929 20:53:44.341072 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.341159 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 20:53:44.342211 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.341765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.3:50823)
0929 20:53:44.341953 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.342034 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 20:53:44.342413 7879 mvcc.cc:205] Tried to move back new op lower bound from 7334769558862209024 to 7334769558572679168. Current Snapshot: MvccSnapshot[applied={T|T < 7334769558862209024}]
I20260929 20:53:44.344985 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.344566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.97.2:39149)
0929 20:53:44.344744 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.344820 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 20:53:44.349203 7301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:44.349434 7301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:44.355260 7301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42034594181341f4b03369c668783273: Sending DeleteTablet for 3 replicas of tablet 39c37e992fd04bacbdf820820b74fabd
I20260929 20:53:44.356168 7562 tablet_service.cc:1514] Processing DeleteTablet for tablet 39c37e992fd04bacbdf820820b74fabd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:44 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:49818
I20260929 20:53:44.356156 7703 tablet_service.cc:1514] Processing DeleteTablet for tablet 39c37e992fd04bacbdf820820b74fabd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:44 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:51752
I20260929 20:53:44.356164 7414 tablet_service.cc:1514] Processing DeleteTablet for tablet 39c37e992fd04bacbdf820820b74fabd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:44 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:44446
I20260929 20:53:44.365564 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 7282
I20260929 20:53:44.368599 7886 tablet_replica.cc:330] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05: stopping tablet replica
I20260929 20:53:44.368844 7886 raft_consensus.cc:2229] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:44.369136 7886 raft_consensus.cc:2258] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:44.370289 7888 tablet_replica.cc:330] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a: stopping tablet replica
I20260929 20:53:44.370342 7886 ts_tablet_manager.cc:1905] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:44.370491 7888 raft_consensus.cc:2229] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:44.370781 7888 raft_consensus.cc:2258] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:44.373451 7888 ts_tablet_manager.cc:1905] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:44.376598 7886 ts_tablet_manager.cc:1918] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:44.376729 7886 log.cc:1196] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/39c37e992fd04bacbdf820820b74fabd
I20260929 20:53:44.377146 7886 ts_tablet_manager.cc:1939] T 39c37e992fd04bacbdf820820b74fabd P 3529304287894649b2750ee4aeb0fa05: Deleting consensus metadata
I20260929 20:53:44.378257 7287 catalog_manager.cc:4841] TS 3529304287894649b2750ee4aeb0fa05 (127.0.97.3:45645): tablet 39c37e992fd04bacbdf820820b74fabd (table test-table [id=bc22e3bc235448e98f2a405ca25f5135]) successfully deleted
I20260929 20:53:44.378373 7887 tablet_replica.cc:330] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91: stopping tablet replica
I20260929 20:53:44.378530 7887 raft_consensus.cc:2229] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:53:44.378885 7887 raft_consensus.cc:2258] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:44.379700 7887 ts_tablet_manager.cc:1905] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:44.379739 7888 ts_tablet_manager.cc:1918] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:44.379840 7888 log.cc:1196] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/39c37e992fd04bacbdf820820b74fabd
I20260929 20:53:44.380230 7888 ts_tablet_manager.cc:1939] T 39c37e992fd04bacbdf820820b74fabd P e816c455c3944cac8973eaac1bf8580a: Deleting consensus metadata
I20260929 20:53:44.381212 7288 catalog_manager.cc:4841] TS e816c455c3944cac8973eaac1bf8580a (127.0.97.1:46367): tablet 39c37e992fd04bacbdf820820b74fabd (table test-table [id=bc22e3bc235448e98f2a405ca25f5135]) successfully deleted
I20260929 20:53:44.384241 7887 ts_tablet_manager.cc:1918] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:44.384338 7887 log.cc:1196] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/39c37e992fd04bacbdf820820b74fabd
I20260929 20:53:44.384712 7887 ts_tablet_manager.cc:1939] T 39c37e992fd04bacbdf820820b74fabd P 9f3bebe27bd744c1bcd36918f642ca91: Deleting consensus metadata
I20260929 20:53:44.385710 7289 catalog_manager.cc:4841] TS 9f3bebe27bd744c1bcd36918f642ca91 (127.0.97.2:36383): tablet 39c37e992fd04bacbdf820820b74fabd (table test-table [id=bc22e3bc235448e98f2a405ca25f5135]) successfully deleted
I20260929 20:53:44.394522 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.366461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44454 (local address 127.0.97.1:46367)
0929 20:53:44.366656 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:44.366662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:44.371328 (+ 4666us) server_negotiation.cc:407] Connection header received
0929 20:53:44.371495 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:44.371500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:44.371574 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:44.371661 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:44.384483 (+ 12822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.385464 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.386763 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.387164 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.387274 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:44.389545 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:44.389558 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:44.389574 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:44.389611 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:44.391987 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:44.392578 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:44.392585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:44.392588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:44.392711 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:44.393062 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:44.393066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:44.393068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:44.393408 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:44.393697 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:44.393966 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 20:53:44.394180 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:53:44.395159 7477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:44.432367 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.431949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.0.1:56344)
0929 20:53:44.432127 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.432212 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 20:53:44.433339 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.433027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34335 (local address 127.0.0.1:56348)
0929 20:53:44.433157 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:44.433228 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 20:53:44.486616 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 7282
I20260929 20:53:44.500244 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 7494
I20260929 20:53:44.513103 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.500884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49836 (local address 127.0.97.2:36383)
0929 20:53:44.501070 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:44.501077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:44.501155 (+ 78us) server_negotiation.cc:407] Connection header received
0929 20:53:44.501336 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:44.501340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:44.501399 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:44.501481 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:44.502797 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.503590 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.504823 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.505194 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.505302 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:44.507737 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:44.507751 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:44.507768 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:44.507805 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:44.510538 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:44.511133 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:44.511141 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:44.511143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:44.511270 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:44.511658 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:44.511663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:44.511667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:44.512013 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:44.512280 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:44.512612 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 20:53:44.512782 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:53:44.513756 7622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:44.628523 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 7494
I20260929 20:53:44.645840 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 7640
I20260929 20:53:44.658296 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:44.646621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51756 (local address 127.0.97.3:45645)
0929 20:53:44.647080 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:44.647087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:44.647125 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:53:44.647221 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:44.647225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:44.647302 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:44.647388 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:44.648632 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.649316 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.650526 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:44.650922 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:44.651026 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:44.653176 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:44.653190 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:44.653206 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:44.653237 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:44.655990 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:44.656574 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:44.656581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:44.656583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:44.656699 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:44.657012 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:44.657016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:44.657018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:44.657373 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:44.657584 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:44.657782 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 20:53:44.657932 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":213,"threads_started":1}
I20260929 20:53:44.658898 7763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:44.748428 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 7640
I20260929 20:53:44.761704 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 6977
I20260929 20:53:44.762725 7145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:44.875465 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 6977
I20260929 20:53:44.891324 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 7077
I20260929 20:53:44.892320 7332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:45.008731 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 7077
2026-09-29T20:53:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9211 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:53:45 dist-test-slave-m9g1 krb5kdc[7898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:53:45 dist-test-slave-m9g1 krb5kdc[7898](info): set up 2 sockets
Sep 29 20:53:45 dist-test-slave-m9g1 krb5kdc[7898](info): commencing operation
krb5kdc: starting...
W20260929 20:53:46.006778 388 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:53:46 dist-test-slave-m9g1 krb5kdc[7898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:53:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:53:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260929 20:53:46.198326 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46269
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:46.501374 7911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:46.501665 7911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:46.501801 7911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:46.508894 7911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:53:46.509007 7911 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 20:53:46.509076 7911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:46.509159 7911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:46.509223 7911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:53:46.509305 7911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:53:46.519019 7911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46269
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:46.520332 7911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:46.521610 7911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:46.532745 7916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:46.532681 7919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:46.532716 7917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:46.533351 7911 server_base.cc:1034] running on GCE node
I20260929 20:53:46.534052 7911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:46.535367 7911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:46.536558 7911 hybrid_clock.cc:648] HybridClock initialized: now 1790715226536494 us; error 49 us; skew 500 ppm
I20260929 20:53:46.536842 7911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:46 dist-test-slave-m9g1 krb5kdc[7898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715226, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:46.545455 7911 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260929 20:53:46.546854 7911 webserver.cc:460] Webserver started at http://127.0.97.62:37837/ using document root <none> and password file <none>
I20260929 20:53:46.547536 7911 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:46.547626 7911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:46.547914 7911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:46.550770 7911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "4d12677949284c699c61ae92f95a6a27"
format_stamp: "Formatted at 2026-09-29 20:53:46 on dist-test-slave-m9g1"
server_key: "e0ff66e4367dbddba16750cce124a1f4"
server_key_iv: "1702ffbd2f0c01348582485f15e20dad"
server_key_version: "encryptionkey@0"
I20260929 20:53:46.551510 7911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "4d12677949284c699c61ae92f95a6a27"
format_stamp: "Formatted at 2026-09-29 20:53:46 on dist-test-slave-m9g1"
server_key: "e0ff66e4367dbddba16750cce124a1f4"
server_key_iv: "1702ffbd2f0c01348582485f15e20dad"
server_key_version: "encryptionkey@0"
I20260929 20:53:46.556300 7911 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 20:53:46.559518 7926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:46.560756 7911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 20:53:46.560902 7911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "4d12677949284c699c61ae92f95a6a27"
format_stamp: "Formatted at 2026-09-29 20:53:46 on dist-test-slave-m9g1"
server_key: "e0ff66e4367dbddba16750cce124a1f4"
server_key_iv: "1702ffbd2f0c01348582485f15e20dad"
server_key_version: "encryptionkey@0"
I20260929 20:53:46.561080 7911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:46.586478 7911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:46.590389 7911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:46.590615 7911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:46.610060 7911 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46269
I20260929 20:53:46.610070 7987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46269 every 8 connection(s)
I20260929 20:53:46.611343 7911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:53:46.615008 7988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:46.615486 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 7911
I20260929 20:53:46.615695 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:53:46.616466 388 external_mini_cluster.cc:1428] Setting key cad54cce1c5797f18b4d7ae6cb0e8bde
I20260929 20:53:46.628546 7988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27: Bootstrap starting.
Sep 29 20:53:46 dist-test-slave-m9g1 krb5kdc[7898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:53:46.632210 7988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:46.633298 7988 log.cc:824] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:46.636294 7988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27: No bootstrap required, opened a new log
I20260929 20:53:46.638926 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:46.618564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.0.97.62:46269)
0929 20:53:46.619358 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:46.619373 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:46.619413 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:53:46.621160 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:46.621184 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:46.621622 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:46.621958 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:46.623638 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:46.624960 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:46.626685 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:46.627354 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:46.627493 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:46.631871 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:46.631878 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:46.631892 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:46.631926 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:46.635165 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:46.635911 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:46.635916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:46.635922 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:46.636083 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:46.636480 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:46.636484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:46.636486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:46.636964 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:46.637187 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:46.637560 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 20:53:46.637849 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":345,"threads_started":1}
I20260929 20:53:46.643630 7988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d12677949284c699c61ae92f95a6a27" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46269 } }
I20260929 20:53:46.643935 7988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:46.644016 7988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d12677949284c699c61ae92f95a6a27, State: Initialized, Role: FOLLOWER
I20260929 20:53:46.644528 7988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d12677949284c699c61ae92f95a6a27" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46269 } }
I20260929 20:53:46.644742 7988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:53:46.644857 7988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:53:46.645012 7988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:46.647025 7988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d12677949284c699c61ae92f95a6a27" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46269 } }
I20260929 20:53:46.647476 7988 leader_election.cc:302] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d12677949284c699c61ae92f95a6a27; no voters:
I20260929 20:53:46.648039 7988 leader_election.cc:288] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:53:46.648121 7993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:46.648891 7993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27 [term 1 LEADER]: Becoming Leader. State: Replica: 4d12677949284c699c61ae92f95a6a27, State: Running, Role: LEADER
I20260929 20:53:46.649369 7993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d12677949284c699c61ae92f95a6a27" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46269 } }
I20260929 20:53:46.649869 7988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:53:46.651968 7994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d12677949284c699c61ae92f95a6a27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d12677949284c699c61ae92f95a6a27" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46269 } } }
I20260929 20:53:46.652415 7994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:46.651995 7995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d12677949284c699c61ae92f95a6a27. Latest consensus state: current_term: 1 leader_uuid: "4d12677949284c699c61ae92f95a6a27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d12677949284c699c61ae92f95a6a27" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46269 } } }
I20260929 20:53:46.653167 7995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27 [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:46.653642 7999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:53:46.660765 7999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:53:46.670320 7999 catalog_manager.cc:1347] Generated new cluster ID: aee41dd06f73453b8ef81170cd4e9ac3
I20260929 20:53:46.670446 7999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:53:46.685379 7999 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:53:46.686697 7999 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:53:46.702622 7999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27: Generated new TSK 0
I20260929 20:53:46.703418 7999 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260929 20:53:46.747769 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46269
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:47.057039 8014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:47.057322 8014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:47.057461 8014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:47.064988 8014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:47.065094 8014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:47.065204 8014 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:53:47.065287 8014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:53:47.074918 8014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:47.076107 8014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:47.077360 8014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:47.089841 8020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:47.089900 8022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:47.089859 8019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:47.090301 8014 server_base.cc:1034] running on GCE node
I20260929 20:53:47.091120 8014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:47.092008 8014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:47.093225 8014 hybrid_clock.cc:648] HybridClock initialized: now 1790715227093162 us; error 43 us; skew 500 ppm
I20260929 20:53:47.093518 8014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:47 dist-test-slave-m9g1 krb5kdc[7898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:47.096400 8014 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 20:53:47.097733 8014 webserver.cc:460] Webserver started at http://127.0.97.1:36763/ using document root <none> and password file <none>
I20260929 20:53:47.098429 8014 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:47.098526 8014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:47.098814 8014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:47.101531 8014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "81f5ae060d2648b5904ebc7f0b756ab5"
format_stamp: "Formatted at 2026-09-29 20:53:47 on dist-test-slave-m9g1"
server_key: "717ea60f42cb5232dc247de93ddbe306"
server_key_iv: "1c52e3fdb1dff6eeedae610aeea4dbfb"
server_key_version: "encryptionkey@0"
I20260929 20:53:47.102402 8014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "81f5ae060d2648b5904ebc7f0b756ab5"
format_stamp: "Formatted at 2026-09-29 20:53:47 on dist-test-slave-m9g1"
server_key: "717ea60f42cb5232dc247de93ddbe306"
server_key_iv: "1c52e3fdb1dff6eeedae610aeea4dbfb"
server_key_version: "encryptionkey@0"
I20260929 20:53:47.107285 8014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:53:47.110548 8029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:47.111771 8014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:53:47.111915 8014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "81f5ae060d2648b5904ebc7f0b756ab5"
format_stamp: "Formatted at 2026-09-29 20:53:47 on dist-test-slave-m9g1"
server_key: "717ea60f42cb5232dc247de93ddbe306"
server_key_iv: "1c52e3fdb1dff6eeedae610aeea4dbfb"
server_key_version: "encryptionkey@0"
I20260929 20:53:47.112113 8014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:47.126650 8014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:47.130910 8014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:47.131160 8014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:47.133687 8014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:47.133783 8014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:47.133908 8014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:47.133973 8014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:53:47 dist-test-slave-m9g1 krb5kdc[7898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:53:47.156896 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:47.134480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44769 (local address 127.0.97.62:46269)
0929 20:53:47.134659 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:47.134666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:47.136057 (+ 1391us) server_negotiation.cc:407] Connection header received
0929 20:53:47.138034 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:47.138040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:47.138125 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:47.138269 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:47.140258 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:47.141138 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:47.144061 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:47.144601 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:47.144698 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:47.149676 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:47.149684 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:47.149690 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:47.149730 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:47.153067 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:47.153995 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:47.154001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:47.154005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:47.154153 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:47.154747 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:47.154753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:47.154756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:47.155121 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:47.155279 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:47.156188 (+ 909us) server_negotiation.cc:299] Negotiation successful
0929 20:53:47.156460 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:53:47.158385 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:47.135108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46269 (local address 127.0.97.1:44769)
0929 20:53:47.135889 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:47.135947 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:47.137746 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:47.138456 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:47.138463 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:47.139125 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:47.140063 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:47.140090 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:47.141294 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:47.141300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:47.143892 (+ 2592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:47.143910 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:47.144848 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:47.144856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:47.145161 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:47.146009 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:47.146040 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:47.149367 (+ 3327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:47.153249 (+ 3882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:47.153259 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:47.153277 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:47.153844 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:47.154282 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:47.154288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:47.154292 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:47.154605 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:47.155237 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:47.155246 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:47.155731 (+ 485us) client_negotiation.cc:769] Sending connection context
0929 20:53:47.156079 (+ 348us) client_negotiation.cc:240] Negotiation successful
0929 20:53:47.156679 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":339,"threads_started":1}
I20260929 20:53:47.172186 8014 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40873
I20260929 20:53:47.172156 8148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40873 every 8 connection(s)
I20260929 20:53:47.173456 8014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:53:47.175413 8149 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46269
I20260929 20:53:47.175632 8149 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:47.176234 8149 heartbeater.cc:511] Master 127.0.97.62:46269 requested a full tablet report, sending...
I20260929 20:53:47.176218 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 8014
I20260929 20:53:47.176397 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:53:47.177157 388 external_mini_cluster.cc:1428] Setting key 5b548c2568e17818f60e57c317f1c92c
I20260929 20:53:47.178261 7943 ts_manager.cc:194] Registered new tserver with Master: 81f5ae060d2648b5904ebc7f0b756ab5 (127.0.97.1:40873)
I20260929 20:53:47.179010 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:47.165368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46269 (local address 127.0.97.1:46389)
0929 20:53:47.166313 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:47.166335 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:47.166521 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:47.166826 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:47.166831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:47.167236 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:47.167499 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:47.167508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:47.168692 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:47.168697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:47.170584 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:47.170597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:47.171559 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:47.171564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:47.171698 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:47.172344 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:47.172366 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:47.173071 (+ 705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:47.176133 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:47.176143 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:47.176146 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:47.176639 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:47.177233 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:47.177238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:47.177241 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:47.177388 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:47.177990 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:47.177995 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:47.178169 (+ 174us) client_negotiation.cc:769] Sending connection context
0929 20:53:47.178324 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 20:53:47.178603 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":642,"threads_started":1}
I20260929 20:53:47.179738 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:47.165814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46389 (local address 127.0.97.62:46269)
0929 20:53:47.165967 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:47.165972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:47.166414 (+ 442us) server_negotiation.cc:407] Connection header received
0929 20:53:47.166646 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:47.166652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:47.166716 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:47.166791 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:47.167628 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:47.168523 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:47.170708 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:47.171186 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:47.171306 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:47.173230 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:47.173237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:47.173241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:47.173272 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:47.175984 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:47.176793 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:47.176798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:47.176802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:47.176944 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:47.177515 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:47.177520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:47.177523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:47.177881 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:47.178035 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:47.179272 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0929 20:53:47.179482 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 20:53:47.179997 7943 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44769
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260929 20:53:47.217554 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46269
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:47.524473 8153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:47.524781 8153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:47.524879 8153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:47.531968 8153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:47.532078 8153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:47.532161 8153 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:53:47.532243 8153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:53:47.541929 8153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:47.543226 8153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:47.544512 8153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:47.556872 8158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:47.556839 8161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:47.556810 8159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:47.557605 8153 server_base.cc:1034] running on GCE node
I20260929 20:53:47.558145 8153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:47.559067 8153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:47.560271 8153 hybrid_clock.cc:648] HybridClock initialized: now 1790715227560206 us; error 41 us; skew 500 ppm
I20260929 20:53:47.560585 8153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:47 dist-test-slave-m9g1 krb5kdc[7898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:47.563537 8153 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 20:53:47.564981 8153 webserver.cc:460] Webserver started at http://127.0.97.2:33761/ using document root <none> and password file <none>
I20260929 20:53:47.565675 8153 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:47.565783 8153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:47.566071 8153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:47.568862 8153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "8a72c22c698049e1923f59f3c839040e"
format_stamp: "Formatted at 2026-09-29 20:53:47 on dist-test-slave-m9g1"
server_key: "394b1bb818ab5aabc8a686963cc26bda"
server_key_iv: "34fd75feb95b6cc67556d48426ceb0ab"
server_key_version: "encryptionkey@0"
I20260929 20:53:47.569633 8153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8a72c22c698049e1923f59f3c839040e"
format_stamp: "Formatted at 2026-09-29 20:53:47 on dist-test-slave-m9g1"
server_key: "394b1bb818ab5aabc8a686963cc26bda"
server_key_iv: "34fd75feb95b6cc67556d48426ceb0ab"
server_key_version: "encryptionkey@0"
I20260929 20:53:47.574805 8153 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 20:53:47.577939 8168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:47.579188 8153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:53:47.579329 8153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "8a72c22c698049e1923f59f3c839040e"
format_stamp: "Formatted at 2026-09-29 20:53:47 on dist-test-slave-m9g1"
server_key: "394b1bb818ab5aabc8a686963cc26bda"
server_key_iv: "34fd75feb95b6cc67556d48426ceb0ab"
server_key_version: "encryptionkey@0"
I20260929 20:53:47.579527 8153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:47.612752 8153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:47.616909 8153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:47.617138 8153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:47.619741 8153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:47.619834 8153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:47.619994 8153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:47.620060 8153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:53:47 dist-test-slave-m9g1 krb5kdc[7898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:53:47.642172 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:47.620828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60901 (local address 127.0.97.62:46269)
0929 20:53:47.620977 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:47.620981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:47.622290 (+ 1309us) server_negotiation.cc:407] Connection header received
0929 20:53:47.625099 (+ 2809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:47.625105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:47.625192 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:47.625264 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:47.627734 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:47.628700 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:47.630754 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:47.631283 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:47.631415 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:47.636268 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:47.636275 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:47.636280 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:47.636321 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:47.639355 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:47.640028 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:47.640033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:47.640037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:47.640157 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:47.640575 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:47.640580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:47.640584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:47.640838 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:47.640972 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:47.641631 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 20:53:47.641855 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 20:53:47.643530 8177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:47.621258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46269 (local address 127.0.97.2:60901)
0929 20:53:47.622118 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:47.622162 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:47.624771 (+ 2609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:47.625802 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:47.625809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:47.626700 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:47.627559 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:47.627588 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:47.628887 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:47.628894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:47.630620 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:47.630635 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:47.631536 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:47.631549 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:47.631883 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:47.632708 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:47.632743 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:47.635964 (+ 3221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:47.639493 (+ 3529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:47.639502 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:47.639519 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:47.639910 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:47.640256 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:47.640261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:47.640265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:47.640466 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:47.640957 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:47.640965 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:47.641297 (+ 332us) client_negotiation.cc:769] Sending connection context
0929 20:53:47.641523 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 20:53:47.642010 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":341,"threads_started":1}
I20260929 20:53:47.662344 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:47.649002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46269 (local address 127.0.97.2:38503)
0929 20:53:47.649558 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:47.649572 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:47.649688 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:47.650002 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:47.650008 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:47.650742 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:47.651235 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:47.651252 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:47.652426 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:47.652431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:47.654289 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:47.654308 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:47.655341 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:47.655349 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:47.655544 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:47.656198 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:47.656221 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:47.657270 (+ 1049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:47.659996 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:47.660002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:47.660006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:47.660353 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:47.660671 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:47.660676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:47.660680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:47.660831 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:47.661349 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:47.661354 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:47.661552 (+ 198us) client_negotiation.cc:769] Sending connection context
0929 20:53:47.661703 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 20:53:47.661983 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":296,"threads_started":1}
I20260929 20:53:47.662472 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:47.648964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38503 (local address 127.0.97.62:46269)
0929 20:53:47.649097 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:47.649100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:47.649642 (+ 542us) server_negotiation.cc:407] Connection header received
0929 20:53:47.649801 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:47.649806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:47.649871 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:47.649950 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:47.651437 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:47.652293 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:47.654448 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:47.654970 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:47.655087 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:47.657444 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:47.657449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:47.657452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:47.657479 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:47.659827 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:47.660458 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:47.660464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:47.660467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:47.660575 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:47.660930 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:47.660935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:47.660939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:47.661244 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:47.661365 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:47.661858 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 20:53:47.662099 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 20:53:47.666733 8153 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40759
I20260929 20:53:47.666752 8287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40759 every 8 connection(s)
I20260929 20:53:47.667868 8153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:53:47.669535 8288 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46269
I20260929 20:53:47.669739 8288 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:47.670354 8288 heartbeater.cc:511] Master 127.0.97.62:46269 requested a full tablet report, sending...
I20260929 20:53:47.671518 7943 ts_manager.cc:194] Registered new tserver with Master: 8a72c22c698049e1923f59f3c839040e (127.0.97.2:40759)
I20260929 20:53:47.672286 7943 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60901
I20260929 20:53:47.677042 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 8153
I20260929 20:53:47.677181 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:53:47.677564 388 external_mini_cluster.cc:1428] Setting key 1361319232817081e28cacbc16e841f0
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 20:53:47.716995 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46269
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:48.021210 8292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:48.021497 8292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:48.021632 8292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:48.028817 8292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:48.028935 8292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:48.029052 8292 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:53:48.029155 8292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:53:48.038895 8292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:48.040202 8292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:48.041692 8292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:48.054273 8298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:48.054342 8297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:48.054311 8300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:48.055048 8292 server_base.cc:1034] running on GCE node
I20260929 20:53:48.055629 8292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:48.056591 8292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:48.057798 8292 hybrid_clock.cc:648] HybridClock initialized: now 1790715228057725 us; error 45 us; skew 500 ppm
I20260929 20:53:48.058080 8292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:48 dist-test-slave-m9g1 krb5kdc[7898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:48.061230 8292 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 20:53:48.062698 8292 webserver.cc:460] Webserver started at http://127.0.97.3:42973/ using document root <none> and password file <none>
I20260929 20:53:48.063370 8292 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:48.063459 8292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:48.063760 8292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:48.066783 8292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "5e2bf586cb8c4807b546b3263c73d17e"
format_stamp: "Formatted at 2026-09-29 20:53:48 on dist-test-slave-m9g1"
server_key: "0fadbb5251bfc7a45009f80cf58ec629"
server_key_iv: "874bfcd45c5a072d4515e83f4e371201"
server_key_version: "encryptionkey@0"
I20260929 20:53:48.067549 8292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5e2bf586cb8c4807b546b3263c73d17e"
format_stamp: "Formatted at 2026-09-29 20:53:48 on dist-test-slave-m9g1"
server_key: "0fadbb5251bfc7a45009f80cf58ec629"
server_key_iv: "874bfcd45c5a072d4515e83f4e371201"
server_key_version: "encryptionkey@0"
I20260929 20:53:48.072588 8292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:53:48.075932 8307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:48.077212 8292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:53:48.077369 8292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "5e2bf586cb8c4807b546b3263c73d17e"
format_stamp: "Formatted at 2026-09-29 20:53:48 on dist-test-slave-m9g1"
server_key: "0fadbb5251bfc7a45009f80cf58ec629"
server_key_iv: "874bfcd45c5a072d4515e83f4e371201"
server_key_version: "encryptionkey@0"
I20260929 20:53:48.077571 8292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:48.094523 8292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:48.098361 8292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:48.098583 8292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:48.100950 8292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:48.101025 8292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:48.101142 8292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:48.101195 8292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:53:48 dist-test-slave-m9g1 krb5kdc[7898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:53:48.123304 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.101521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57969 (local address 127.0.97.62:46269)
0929 20:53:48.101709 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:48.101715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:48.103050 (+ 1335us) server_negotiation.cc:407] Connection header received
0929 20:53:48.104976 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:48.104981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:48.105063 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:48.105161 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:48.107526 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.108452 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.110620 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.111180 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.111300 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.116013 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:48.116020 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:48.116025 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:48.116063 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:48.119551 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:48.120555 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:48.120561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:48.120565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:48.120752 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:48.121305 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:48.121310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:48.121314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:48.121683 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:48.121848 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:48.122751 (+ 903us) server_negotiation.cc:299] Negotiation successful
0929 20:53:48.122976 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:53:48.124366 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.102033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46269 (local address 127.0.97.3:57969)
0929 20:53:48.102872 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:48.102918 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:48.104720 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:48.105562 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:48.105577 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:48.106410 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:48.107327 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:48.107356 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.108583 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.108589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:48.110481 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:48.110503 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.111448 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.111456 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:48.111769 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.112547 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:48.112584 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:48.115769 (+ 3185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:48.119701 (+ 3932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:48.119708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:48.119720 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:48.120406 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:48.120876 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:48.120881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:48.120886 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:48.121111 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:48.121831 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:48.121839 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:48.122332 (+ 493us) client_negotiation.cc:769] Sending connection context
0929 20:53:48.122637 (+ 305us) client_negotiation.cc:240] Negotiation successful
0929 20:53:48.123197 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":355,"threads_started":1}
I20260929 20:53:48.142549 8292 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35543
I20260929 20:53:48.142562 8426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35543 every 8 connection(s)
I20260929 20:53:48.143850 8292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:53:48.144006 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.130617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53763 (local address 127.0.97.62:46269)
0929 20:53:48.130803 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:48.130809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:48.131420 (+ 611us) server_negotiation.cc:407] Connection header received
0929 20:53:48.131697 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:48.131703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:48.131787 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:48.131893 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:48.132701 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.133842 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.135333 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.135982 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.136112 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.138567 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:48.138574 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:48.138579 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:48.138616 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:48.141470 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:48.142228 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:48.142233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:48.142235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:48.142369 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:48.142700 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:48.142705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:48.142707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:48.142935 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:48.143072 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:48.143454 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 20:53:48.143636 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:53:48.144096 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.130623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46269 (local address 127.0.97.3:53763)
0929 20:53:48.131324 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:48.131343 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:48.131530 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:48.131912 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:48.131915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:48.132324 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:48.132587 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:48.132595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.133973 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.133976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:48.135188 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:48.135200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.136171 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.136178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:48.136365 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.137004 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:48.137035 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:48.138342 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:48.141584 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:48.141589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:48.141594 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:48.142086 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:48.142472 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:48.142477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:48.142480 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:48.142597 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:48.143036 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:48.143040 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:48.143216 (+ 176us) client_negotiation.cc:769] Sending connection context
0929 20:53:48.143328 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 20:53:48.143632 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":415,"threads_started":1}
I20260929 20:53:48.145298 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 8292
I20260929 20:53:48.145438 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:53:48.145861 388 external_mini_cluster.cc:1428] Setting key 258791787b95ed8e7a23d226dfa4ec03
I20260929 20:53:48.148324 8427 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46269
I20260929 20:53:48.148533 8427 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:48.149225 8427 heartbeater.cc:511] Master 127.0.97.62:46269 requested a full tablet report, sending...
I20260929 20:53:48.150523 7943 ts_manager.cc:194] Registered new tserver with Master: 5e2bf586cb8c4807b546b3263c73d17e (127.0.97.3:35543)
I20260929 20:53:48.151371 7943 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57969
I20260929 20:53:48.161629 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:53:48.182874 8149 heartbeater.cc:503] Master 127.0.97.62:46269 was elected leader, sending a full tablet report...
Sep 29 20:53:48 dist-test-slave-m9g1 krb5kdc[7898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715228, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:53:48 dist-test-slave-m9g1 krb5kdc[7898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:53:48 dist-test-slave-m9g1 krb5kdc[7898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:53:48.207309 7991 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:48.198043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51384 (local address 127.0.97.62:46269)
0929 20:53:48.198276 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:48.198282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:48.198461 (+ 179us) server_negotiation.cc:407] Connection header received
0929 20:53:48.198599 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:48.198603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:48.198665 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:48.198744 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:48.199911 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.200665 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.201862 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.202291 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.202373 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.207111 (+ 4738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51384: BlockingRecv error: recv got EOF from 127.0.0.1:51384 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 20:53:48 dist-test-slave-m9g1 krb5kdc[7898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715228, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:53:48.221769 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.210125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51388 (local address 127.0.97.62:46269)
0929 20:53:48.210337 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:48.210342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:48.210552 (+ 210us) server_negotiation.cc:407] Connection header received
0929 20:53:48.210679 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:48.210683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:48.210754 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:48.210830 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:48.211884 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.212551 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.213644 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.214028 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.214126 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.217476 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:48.217481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:48.217484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:48.217515 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:48.219739 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:48.220299 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:48.220302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:48.220305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:48.220412 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:48.220709 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:48.220713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:48.220715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:48.220983 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:48.221173 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:48.221365 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 20:53:48.221517 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 20:53:48.225512 7943 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:53:48.232246 7943 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:53:48.262806 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.245062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43798 (local address 127.0.97.3:35543)
0929 20:53:48.255985 (+ 10923us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:48.255993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:48.256023 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:53:48.256132 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:48.256139 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:48.256334 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:48.256515 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:48.257124 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.258526 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.260490 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.261663 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.261786 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.262017 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:48.262312 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 20:53:48.262464 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10742,"thread_start_us":11126,"threads_started":1}
I20260929 20:53:48.262864 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.244726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35543 (local address 127.0.0.1:43798)
0929 20:53:48.254012 (+ 9286us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:48.254031 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:48.254164 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:48.256510 (+ 2346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:48.256515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:48.256537 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:48.256895 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:48.256908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.258650 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.258654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:48.260323 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:48.260334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.261879 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.261885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:48.262069 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.262116 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 20:53:48.262267 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 20:53:48.262460 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9164,"mutex_wait_us":22,"thread_start_us":8669,"threads_started":1}
I20260929 20:53:48.263391 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.244618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39964 (local address 127.0.97.2:40759)
0929 20:53:48.254689 (+ 10071us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:48.254699 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:48.254731 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:53:48.254849 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:48.254858 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:48.255138 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:48.255384 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:48.256002 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.257953 (+ 1951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.260491 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.261803 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.261923 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.262102 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:48.262955 (+ 853us) server_negotiation.cc:299] Negotiation successful
0929 20:53:48.263115 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9900,"thread_start_us":9754,"threads_started":1}
I20260929 20:53:48.263454 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.243532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40873 (local address 127.0.0.1:35030)
0929 20:53:48.244355 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:48.244401 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:48.244561 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:48.255615 (+ 11054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:48.255620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:48.255649 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:48.256088 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:48.256100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.257783 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.257787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:48.260543 (+ 2756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:48.260554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.262682 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.262687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:48.262906 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.262935 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 20:53:48.263011 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 20:53:48.263124 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":241,"thread_start_us":429,"threads_started":1}
I20260929 20:53:48.263769 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.244150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40759 (local address 127.0.0.1:39964)
0929 20:53:48.253681 (+ 9531us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:48.253702 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:48.253833 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:48.255389 (+ 1556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:48.255396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:48.255427 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:48.255839 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:48.255849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.258096 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.258101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:48.260324 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:48.260336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.262276 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.262282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:48.262435 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.262465 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:53:48.262542 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 20:53:48.262680 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9387,"mutex_wait_us":80,"thread_start_us":9561,"threads_started":1}
I20260929 20:53:48.264390 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.243737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.0.97.1:40873)
0929 20:53:48.255006 (+ 11269us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:48.255012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:48.255033 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:53:48.255109 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:48.255114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:48.255294 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:48.255482 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:48.256251 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.257658 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.260818 (+ 3160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.262263 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.262378 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.262547 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:48.263963 (+ 1416us) server_negotiation.cc:299] Negotiation successful
0929 20:53:48.264130 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11145,"thread_start_us":11012,"threads_started":1}
I20260929 20:53:48.267311 8218 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4aebdbed2f4db5936512fbe4363d8f (DEFAULT_TABLE table=test-table [id=34e842ae776741f89c38b591d6af02bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:48.267176 8357 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4aebdbed2f4db5936512fbe4363d8f (DEFAULT_TABLE table=test-table [id=34e842ae776741f89c38b591d6af02bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:48.267653 8079 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4aebdbed2f4db5936512fbe4363d8f (DEFAULT_TABLE table=test-table [id=34e842ae776741f89c38b591d6af02bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:48.268425 8218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c4aebdbed2f4db5936512fbe4363d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:48.268549 8357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c4aebdbed2f4db5936512fbe4363d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:48.268795 8079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c4aebdbed2f4db5936512fbe4363d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:48.279116 8451 tablet_bootstrap.cc:492] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e: Bootstrap starting.
I20260929 20:53:48.282423 8453 tablet_bootstrap.cc:492] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e: Bootstrap starting.
I20260929 20:53:48.283473 8451 tablet_bootstrap.cc:654] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:48.284778 8451 log.cc:824] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:48.285755 8453 tablet_bootstrap.cc:654] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:48.285769 8452 tablet_bootstrap.cc:492] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5: Bootstrap starting.
I20260929 20:53:48.286957 8453 log.cc:824] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:48.288870 8452 tablet_bootstrap.cc:654] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:48.289799 8453 tablet_bootstrap.cc:492] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e: No bootstrap required, opened a new log
I20260929 20:53:48.290015 8452 log.cc:824] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:48.290066 8453 ts_tablet_manager.cc:1397] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260929 20:53:48.290323 8451 tablet_bootstrap.cc:492] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e: No bootstrap required, opened a new log
I20260929 20:53:48.290565 8451 ts_tablet_manager.cc:1397] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260929 20:53:48.292748 8452 tablet_bootstrap.cc:492] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5: No bootstrap required, opened a new log
I20260929 20:53:48.292977 8452 ts_tablet_manager.cc:1397] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260929 20:53:48.297134 8453 raft_consensus.cc:348] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } }
I20260929 20:53:48.297454 8453 raft_consensus.cc:374] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:48.297538 8453 raft_consensus.cc:729] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e2bf586cb8c4807b546b3263c73d17e, State: Initialized, Role: FOLLOWER
I20260929 20:53:48.298061 8453 consensus_queue.cc:260] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } }
I20260929 20:53:48.299083 8427 heartbeater.cc:503] Master 127.0.97.62:46269 was elected leader, sending a full tablet report...
I20260929 20:53:48.299678 8453 ts_tablet_manager.cc:1428] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260929 20:53:48.300673 8451 raft_consensus.cc:348] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } }
I20260929 20:53:48.301072 8451 raft_consensus.cc:374] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:48.301163 8451 raft_consensus.cc:729] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a72c22c698049e1923f59f3c839040e, State: Initialized, Role: FOLLOWER
I20260929 20:53:48.301717 8451 consensus_queue.cc:260] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } }
I20260929 20:53:48.302768 8452 raft_consensus.cc:348] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } }
I20260929 20:53:48.303149 8452 raft_consensus.cc:374] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:48.303117 8288 heartbeater.cc:503] Master 127.0.97.62:46269 was elected leader, sending a full tablet report...
I20260929 20:53:48.303236 8452 raft_consensus.cc:729] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f5ae060d2648b5904ebc7f0b756ab5, State: Initialized, Role: FOLLOWER
I20260929 20:53:48.303716 8451 ts_tablet_manager.cc:1428] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260929 20:53:48.303843 8452 consensus_queue.cc:260] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } }
I20260929 20:53:48.305186 8452 ts_tablet_manager.cc:1428] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
W20260929 20:53:48.398438 8428 tablet.cc:2369] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:53:48.419653 8289 tablet.cc:2369] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:53:48.425699 8150 tablet.cc:2369] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:53:48.489689 8457 raft_consensus.cc:482] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:53:48.489948 8457 raft_consensus.cc:504] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } }
I20260929 20:53:48.491600 8457 leader_election.cc:288] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81f5ae060d2648b5904ebc7f0b756ab5 (127.0.97.1:40873), 8a72c22c698049e1923f59f3c839040e (127.0.97.2:40759)
I20260929 20:53:48.496770 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.491913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40759 (local address 127.0.97.3:43275)
0929 20:53:48.492110 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:48.492129 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:48.492245 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:48.492535 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:48.492540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:48.492574 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:48.492765 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:48.492773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.493745 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.493748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:48.495077 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:48.495088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.496107 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.496113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:48.496277 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.496300 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:53:48.496378 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 20:53:48.496497 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":40}
I20260929 20:53:48.496775 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.492054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43275 (local address 127.0.97.2:40759)
0929 20:53:48.492236 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:48.492241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:48.492256 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:53:48.492362 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:48.492366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:48.492420 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:48.492499 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:48.492891 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.493614 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.495240 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.495967 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.496083 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.496184 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:48.496413 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 20:53:48.496534 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:53:48.497664 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.491913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40873 (local address 127.0.97.3:59583)
0929 20:53:48.492792 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:48.492808 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:48.492911 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:48.493196 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:48.493200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:48.493222 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:48.493531 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:48.493540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.494497 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.494502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:48.495812 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:48.495822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.497000 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.497008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:48.497210 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.497242 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:53:48.497326 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 20:53:48.497411 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":201,"thread_start_us":360,"threads_started":1}
I20260929 20:53:48.497718 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.491926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59583 (local address 127.0.97.1:40873)
0929 20:53:48.492105 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:48.492111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:48.492896 (+ 785us) server_negotiation.cc:407] Connection header received
0929 20:53:48.493032 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:48.493036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:48.493101 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:48.493182 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:48.493673 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.494382 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.495948 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.496674 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.496786 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.496887 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:48.497385 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 20:53:48.497507 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:53:48.497812 8238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4aebdbed2f4db5936512fbe4363d8f" candidate_uuid: "5e2bf586cb8c4807b546b3263c73d17e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a72c22c698049e1923f59f3c839040e" is_pre_election: true
I20260929 20:53:48.498340 8238 raft_consensus.cc:2454] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e2bf586cb8c4807b546b3263c73d17e in term 0.
I20260929 20:53:48.498569 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4aebdbed2f4db5936512fbe4363d8f" candidate_uuid: "5e2bf586cb8c4807b546b3263c73d17e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" is_pre_election: true
I20260929 20:53:48.498917 8309 leader_election.cc:302] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e2bf586cb8c4807b546b3263c73d17e, 8a72c22c698049e1923f59f3c839040e; no voters:
I20260929 20:53:48.499058 8099 raft_consensus.cc:2454] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e2bf586cb8c4807b546b3263c73d17e in term 0.
I20260929 20:53:48.499325 8457 raft_consensus.cc:2781] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:53:48.499468 8457 raft_consensus.cc:482] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:53:48.499568 8457 raft_consensus.cc:3037] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:48.501654 8457 raft_consensus.cc:504] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } }
I20260929 20:53:48.502229 8457 leader_election.cc:288] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [CANDIDATE]: Term 1 election: Requested vote from peers 81f5ae060d2648b5904ebc7f0b756ab5 (127.0.97.1:40873), 8a72c22c698049e1923f59f3c839040e (127.0.97.2:40759)
I20260929 20:53:48.502497 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4aebdbed2f4db5936512fbe4363d8f" candidate_uuid: "5e2bf586cb8c4807b546b3263c73d17e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f5ae060d2648b5904ebc7f0b756ab5"
I20260929 20:53:48.502620 8238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4aebdbed2f4db5936512fbe4363d8f" candidate_uuid: "5e2bf586cb8c4807b546b3263c73d17e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a72c22c698049e1923f59f3c839040e"
I20260929 20:53:48.502686 8099 raft_consensus.cc:3037] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:48.502800 8238 raft_consensus.cc:3037] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:48.505666 8238 raft_consensus.cc:2454] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e2bf586cb8c4807b546b3263c73d17e in term 1.
I20260929 20:53:48.505666 8099 raft_consensus.cc:2454] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e2bf586cb8c4807b546b3263c73d17e in term 1.
I20260929 20:53:48.505990 8309 leader_election.cc:302] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e2bf586cb8c4807b546b3263c73d17e, 8a72c22c698049e1923f59f3c839040e; no voters:
I20260929 20:53:48.506237 8457 raft_consensus.cc:2781] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:48.506636 8457 raft_consensus.cc:686] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [term 1 LEADER]: Becoming Leader. State: Replica: 5e2bf586cb8c4807b546b3263c73d17e, State: Running, Role: LEADER
I20260929 20:53:48.507092 8457 consensus_queue.cc:237] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } }
I20260929 20:53:48.511049 7943 catalog_manager.cc:5495] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e2bf586cb8c4807b546b3263c73d17e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "5e2bf586cb8c4807b546b3263c73d17e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } health_report { overall_health: UNKNOWN } } }
I20260929 20:53:48.555280 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.550714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43802 (local address 127.0.97.3:35543)
0929 20:53:48.550909 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:48.550913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:48.550930 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:53:48.551067 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:48.551071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:48.551150 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:48.551274 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:48.551537 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.552199 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.553439 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.553823 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.553911 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.554298 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:48.554808 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:48.554910 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:48.554964 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 20:53:48.555023 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:53:48.562656 8238 raft_consensus.cc:1261] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e [term 1 FOLLOWER]: Refusing update from remote peer 5e2bf586cb8c4807b546b3263c73d17e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:48.562655 8099 raft_consensus.cc:1261] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 FOLLOWER]: Refusing update from remote peer 5e2bf586cb8c4807b546b3263c73d17e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:48.563444 8461 consensus_queue.cc:1035] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [LEADER]: Connected to new peer: Peer: permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:48.563758 8457 consensus_queue.cc:1035] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:48.576609 8466 mvcc.cc:205] Tried to move back new op lower bound from 7334769576184295424 to 7334769575968628736. Current Snapshot: MvccSnapshot[applied={T|T < 7334769576184295424}]
I20260929 20:53:48.579947 8464 mvcc.cc:205] Tried to move back new op lower bound from 7334769576184295424 to 7334769575968628736. Current Snapshot: MvccSnapshot[applied={T|T < 7334769576184295424}]
I20260929 20:53:48.594334 7942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:48.594660 7942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:48.598368 7942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27: Sending DeleteTablet for 3 replicas of tablet 0c4aebdbed2f4db5936512fbe4363d8f
I20260929 20:53:48.599642 8218 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c4aebdbed2f4db5936512fbe4363d8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:48 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39964
I20260929 20:53:48.599818 8357 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c4aebdbed2f4db5936512fbe4363d8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:48 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43798
I20260929 20:53:48.600340 8079 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c4aebdbed2f4db5936512fbe4363d8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:48 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35030
I20260929 20:53:48.600683 7942 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:53:48.601500 7942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:53:48.603842 8475 tablet_replica.cc:330] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e: stopping tablet replica
I20260929 20:53:48.604192 8474 tablet_replica.cc:330] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e: stopping tablet replica
I20260929 20:53:48.604372 8475 raft_consensus.cc:2229] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:53:48.604701 8474 raft_consensus.cc:2229] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:48.605167 8474 raft_consensus.cc:2258] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:48.605144 8475 raft_consensus.cc:2258] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:48.605669 8476 tablet_replica.cc:330] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5: stopping tablet replica
I20260929 20:53:48.606353 8476 raft_consensus.cc:2229] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:48.606874 8476 raft_consensus.cc:2258] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:48.608321 8474 ts_tablet_manager.cc:1905] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:48.608817 8476 ts_tablet_manager.cc:1905] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:48.609256 8079 tablet_service.cc:1467] Processing CreateTablet for tablet 3a39bf5ae6d84f45ac35fa39e6aff769 (DEFAULT_TABLE table=test-table [id=c153c13c5fe34340b1e9083d0a2e08f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:48.609664 8079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a39bf5ae6d84f45ac35fa39e6aff769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:48.610056 8475 ts_tablet_manager.cc:1905] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:48.610525 8357 tablet_service.cc:1467] Processing CreateTablet for tablet 3a39bf5ae6d84f45ac35fa39e6aff769 (DEFAULT_TABLE table=test-table [id=c153c13c5fe34340b1e9083d0a2e08f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:48.610921 8357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a39bf5ae6d84f45ac35fa39e6aff769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:48.614650 8218 tablet_service.cc:1467] Processing CreateTablet for tablet 3a39bf5ae6d84f45ac35fa39e6aff769 (DEFAULT_TABLE table=test-table [id=c153c13c5fe34340b1e9083d0a2e08f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:48.615893 8476 ts_tablet_manager.cc:1918] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:48.616124 8476 log.cc:1196] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/0c4aebdbed2f4db5936512fbe4363d8f
I20260929 20:53:48.616703 8476 ts_tablet_manager.cc:1939] T 0c4aebdbed2f4db5936512fbe4363d8f P 81f5ae060d2648b5904ebc7f0b756ab5: Deleting consensus metadata
I20260929 20:53:48.617080 8475 ts_tablet_manager.cc:1918] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:48.617229 8475 log.cc:1196] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/0c4aebdbed2f4db5936512fbe4363d8f
I20260929 20:53:48.617655 8475 ts_tablet_manager.cc:1939] T 0c4aebdbed2f4db5936512fbe4363d8f P 5e2bf586cb8c4807b546b3263c73d17e: Deleting consensus metadata
I20260929 20:53:48.618857 7927 catalog_manager.cc:4841] TS 81f5ae060d2648b5904ebc7f0b756ab5 (127.0.97.1:40873): tablet 0c4aebdbed2f4db5936512fbe4363d8f (table test-table [id=34e842ae776741f89c38b591d6af02bf]) successfully deleted
I20260929 20:53:48.619143 7927 catalog_manager.cc:4841] TS 5e2bf586cb8c4807b546b3263c73d17e (127.0.97.3:35543): tablet 0c4aebdbed2f4db5936512fbe4363d8f (table test-table [id=34e842ae776741f89c38b591d6af02bf]) successfully deleted
I20260929 20:53:48.624452 8452 tablet_bootstrap.cc:492] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5: Bootstrap starting.
I20260929 20:53:48.624864 8453 tablet_bootstrap.cc:492] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e: Bootstrap starting.
I20260929 20:53:48.626111 8452 tablet_bootstrap.cc:654] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:48.626549 8453 tablet_bootstrap.cc:654] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:48.629994 8452 tablet_bootstrap.cc:492] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5: No bootstrap required, opened a new log
I20260929 20:53:48.630139 8452 ts_tablet_manager.cc:1397] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 20:53:48.630935 8452 raft_consensus.cc:348] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:48.631165 8452 raft_consensus.cc:374] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:48.631256 8452 raft_consensus.cc:729] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f5ae060d2648b5904ebc7f0b756ab5, State: Initialized, Role: FOLLOWER
I20260929 20:53:48.631352 8218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a39bf5ae6d84f45ac35fa39e6aff769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:48.631480 8452 consensus_queue.cc:260] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:48.634461 8452 ts_tablet_manager.cc:1428] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260929 20:53:48.639081 8451 tablet_bootstrap.cc:492] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e: Bootstrap starting.
I20260929 20:53:48.640969 8451 tablet_bootstrap.cc:654] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:48.642719 8453 tablet_bootstrap.cc:492] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e: No bootstrap required, opened a new log
I20260929 20:53:48.642885 8453 ts_tablet_manager.cc:1397] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20260929 20:53:48.643723 8453 raft_consensus.cc:348] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:48.643955 8453 raft_consensus.cc:374] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:48.644048 8453 raft_consensus.cc:729] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e2bf586cb8c4807b546b3263c73d17e, State: Initialized, Role: FOLLOWER
I20260929 20:53:48.644362 8453 consensus_queue.cc:260] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:48.645004 8453 ts_tablet_manager.cc:1428] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 20:53:48.645608 8474 ts_tablet_manager.cc:1918] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:48.645821 8474 log.cc:1196] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/0c4aebdbed2f4db5936512fbe4363d8f
I20260929 20:53:48.646385 8474 ts_tablet_manager.cc:1939] T 0c4aebdbed2f4db5936512fbe4363d8f P 8a72c22c698049e1923f59f3c839040e: Deleting consensus metadata
I20260929 20:53:48.648976 7928 catalog_manager.cc:4841] TS 8a72c22c698049e1923f59f3c839040e (127.0.97.2:40759): tablet 0c4aebdbed2f4db5936512fbe4363d8f (table test-table [id=34e842ae776741f89c38b591d6af02bf]) successfully deleted
I20260929 20:53:48.650314 8451 tablet_bootstrap.cc:492] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e: No bootstrap required, opened a new log
I20260929 20:53:48.650437 8451 ts_tablet_manager.cc:1397] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260929 20:53:48.651057 8451 raft_consensus.cc:348] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:48.651225 8451 raft_consensus.cc:374] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:48.651312 8451 raft_consensus.cc:729] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a72c22c698049e1923f59f3c839040e, State: Initialized, Role: FOLLOWER
I20260929 20:53:48.651535 8451 consensus_queue.cc:260] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:48.652024 8451 ts_tablet_manager.cc:1428] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 20:53:48.939769 8458 raft_consensus.cc:482] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:53:48.940016 8458 raft_consensus.cc:504] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:48.941565 8458 leader_election.cc:288] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81f5ae060d2648b5904ebc7f0b756ab5 (127.0.97.1:40873), 5e2bf586cb8c4807b546b3263c73d17e (127.0.97.3:35543)
I20260929 20:53:48.948958 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.941968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54509 (local address 127.0.97.1:40873)
0929 20:53:48.942171 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:48.942178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:48.942603 (+ 425us) server_negotiation.cc:407] Connection header received
0929 20:53:48.942929 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:48.942935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:48.943014 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:48.943079 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:48.943884 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.944981 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.946998 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.947971 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.948097 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.948195 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:48.948561 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 20:53:48.948692 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:53:48.949278 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.941898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40873 (local address 127.0.97.2:54509)
0929 20:53:48.942489 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:48.942510 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:48.942738 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:48.943280 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:48.943285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:48.943316 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:48.943669 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:48.943684 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.945094 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.945099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:48.946860 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:48.946871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.948176 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.948183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:48.948346 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.948382 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:53:48.948521 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 20:53:48.948658 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":23,"spinlock_wait_cycles":42496,"thread_start_us":293,"threads_started":1}
I20260929 20:53:48.950001 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a39bf5ae6d84f45ac35fa39e6aff769" candidate_uuid: "8a72c22c698049e1923f59f3c839040e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" is_pre_election: true
I20260929 20:53:48.950117 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.942585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35543 (local address 127.0.97.2:54471)
0929 20:53:48.943318 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:48.943332 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:48.943482 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:48.943763 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:48.943767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:48.943789 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:48.944108 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:48.944117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.945319 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.945324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:48.947460 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:48.947474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.949345 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.949352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:48.949546 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.949579 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:53:48.949663 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 20:53:48.949776 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"spinlock_wait_cycles":896,"thread_start_us":474,"threads_started":1}
I20260929 20:53:48.950121 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:48.942718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54471 (local address 127.0.97.3:35543)
0929 20:53:48.942892 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:48.942898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:48.943416 (+ 518us) server_negotiation.cc:407] Connection header received
0929 20:53:48.943607 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:48.943611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:48.943665 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:48.943741 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:48.944239 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.945182 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.947621 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:48.948773 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:48.948881 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:48.949027 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:48.949706 (+ 679us) server_negotiation.cc:299] Negotiation successful
0929 20:53:48.949838 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:53:48.950279 8099 raft_consensus.cc:2454] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a72c22c698049e1923f59f3c839040e in term 0.
I20260929 20:53:48.950819 8169 leader_election.cc:302] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81f5ae060d2648b5904ebc7f0b756ab5, 8a72c22c698049e1923f59f3c839040e; no voters:
I20260929 20:53:48.950919 8377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a39bf5ae6d84f45ac35fa39e6aff769" candidate_uuid: "8a72c22c698049e1923f59f3c839040e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e2bf586cb8c4807b546b3263c73d17e" is_pre_election: true
I20260929 20:53:48.951257 8458 raft_consensus.cc:2781] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:53:48.951365 8458 raft_consensus.cc:482] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:53:48.951350 8377 raft_consensus.cc:2454] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a72c22c698049e1923f59f3c839040e in term 0.
I20260929 20:53:48.951442 8458 raft_consensus.cc:3037] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:48.953605 8458 raft_consensus.cc:504] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:48.954178 8458 leader_election.cc:288] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [CANDIDATE]: Term 1 election: Requested vote from peers 81f5ae060d2648b5904ebc7f0b756ab5 (127.0.97.1:40873), 5e2bf586cb8c4807b546b3263c73d17e (127.0.97.3:35543)
I20260929 20:53:48.954582 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a39bf5ae6d84f45ac35fa39e6aff769" candidate_uuid: "8a72c22c698049e1923f59f3c839040e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f5ae060d2648b5904ebc7f0b756ab5"
I20260929 20:53:48.954752 8099 raft_consensus.cc:3037] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:48.954733 8377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a39bf5ae6d84f45ac35fa39e6aff769" candidate_uuid: "8a72c22c698049e1923f59f3c839040e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e2bf586cb8c4807b546b3263c73d17e"
I20260929 20:53:48.954937 8377 raft_consensus.cc:3037] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:48.957056 8099 raft_consensus.cc:2454] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a72c22c698049e1923f59f3c839040e in term 1.
I20260929 20:53:48.957393 8169 leader_election.cc:302] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81f5ae060d2648b5904ebc7f0b756ab5, 8a72c22c698049e1923f59f3c839040e; no voters:
I20260929 20:53:48.957654 8458 raft_consensus.cc:2781] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:48.958086 8458 raft_consensus.cc:686] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [term 1 LEADER]: Becoming Leader. State: Replica: 8a72c22c698049e1923f59f3c839040e, State: Running, Role: LEADER
I20260929 20:53:48.958173 8377 raft_consensus.cc:2454] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a72c22c698049e1923f59f3c839040e in term 1.
I20260929 20:53:48.958810 8458 consensus_queue.cc:237] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:48.962288 7943 catalog_manager.cc:5495] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a72c22c698049e1923f59f3c839040e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8a72c22c698049e1923f59f3c839040e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } health_report { overall_health: UNKNOWN } } }
Sep 29 20:53:49 dist-test-slave-m9g1 krb5kdc[7898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715226, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:53:49.024468 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.012428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46269 (local address 127.0.0.1:51394)
0929 20:53:49.013057 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:49.013074 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:49.013204 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:49.013517 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:49.013520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:49.014152 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:49.014372 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:49.014381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.015460 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.015463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:49.016399 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:49.016410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.017035 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.017040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:49.017165 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.017719 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:49.017738 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:49.019764 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:49.022523 (+ 2759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:49.022532 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:49.022537 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:49.022898 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:49.023179 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:49.023182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:49.023184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:49.023303 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:49.023716 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:49.023722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:49.023854 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 20:53:49.023990 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 20:53:49.024191 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":337,"threads_started":1}
I20260929 20:53:49.024592 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.012498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.0.97.62:46269)
0929 20:53:49.013110 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:49.013114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:49.013135 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:53:49.013298 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:49.013301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:49.013367 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:49.013454 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:49.014502 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.015323 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.016508 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.016901 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.016988 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.019935 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:49.019940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:49.019944 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:49.019974 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:49.022409 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:49.022984 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:49.022987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:49.022990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:49.023099 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:49.023378 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:49.023381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:49.023383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:49.023622 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:49.023729 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:49.024070 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 20:53:49.024299 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":526,"thread_start_us":368,"threads_started":1}
I20260929 20:53:49.029553 7943 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51394:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 20:53:49.030588 7943 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:53:49.036751 8079 tablet_service.cc:1467] Processing CreateTablet for tablet 3455c9e265e4420698d9478a7c2efee0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70074cd688894cd281fa4d287ec79a36]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:53:49.036751 8218 tablet_service.cc:1467] Processing CreateTablet for tablet 3455c9e265e4420698d9478a7c2efee0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70074cd688894cd281fa4d287ec79a36]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:53:49.036904 8357 tablet_service.cc:1467] Processing CreateTablet for tablet 3455c9e265e4420698d9478a7c2efee0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70074cd688894cd281fa4d287ec79a36]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:53:49.037285 8218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3455c9e265e4420698d9478a7c2efee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:49.037299 8079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3455c9e265e4420698d9478a7c2efee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:49.037372 8357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3455c9e265e4420698d9478a7c2efee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:49.042824 8451 tablet_bootstrap.cc:492] T 3455c9e265e4420698d9478a7c2efee0 P 8a72c22c698049e1923f59f3c839040e: Bootstrap starting.
I20260929 20:53:49.043955 8453 tablet_bootstrap.cc:492] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e: Bootstrap starting.
I20260929 20:53:49.044147 8451 tablet_bootstrap.cc:654] T 3455c9e265e4420698d9478a7c2efee0 P 8a72c22c698049e1923f59f3c839040e: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:49.044166 8452 tablet_bootstrap.cc:492] T 3455c9e265e4420698d9478a7c2efee0 P 81f5ae060d2648b5904ebc7f0b756ab5: Bootstrap starting.
I20260929 20:53:49.045454 8452 tablet_bootstrap.cc:654] T 3455c9e265e4420698d9478a7c2efee0 P 81f5ae060d2648b5904ebc7f0b756ab5: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:49.045516 8453 tablet_bootstrap.cc:654] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:49.047060 8451 tablet_bootstrap.cc:492] T 3455c9e265e4420698d9478a7c2efee0 P 8a72c22c698049e1923f59f3c839040e: No bootstrap required, opened a new log
I20260929 20:53:49.047206 8451 ts_tablet_manager.cc:1397] T 3455c9e265e4420698d9478a7c2efee0 P 8a72c22c698049e1923f59f3c839040e: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260929 20:53:49.047725 8452 tablet_bootstrap.cc:492] T 3455c9e265e4420698d9478a7c2efee0 P 81f5ae060d2648b5904ebc7f0b756ab5: No bootstrap required, opened a new log
I20260929 20:53:49.047828 8453 tablet_bootstrap.cc:492] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e: No bootstrap required, opened a new log
I20260929 20:53:49.047871 8452 ts_tablet_manager.cc:1397] T 3455c9e265e4420698d9478a7c2efee0 P 81f5ae060d2648b5904ebc7f0b756ab5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260929 20:53:49.047983 8453 ts_tablet_manager.cc:1397] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260929 20:53:49.047933 8451 raft_consensus.cc:348] T 3455c9e265e4420698d9478a7c2efee0 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.048158 8451 raft_consensus.cc:374] T 3455c9e265e4420698d9478a7c2efee0 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:49.048234 8451 raft_consensus.cc:729] T 3455c9e265e4420698d9478a7c2efee0 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a72c22c698049e1923f59f3c839040e, State: Initialized, Role: FOLLOWER
I20260929 20:53:49.048473 8451 consensus_queue.cc:260] T 3455c9e265e4420698d9478a7c2efee0 P 8a72c22c698049e1923f59f3c839040e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.048521 8452 raft_consensus.cc:348] T 3455c9e265e4420698d9478a7c2efee0 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.048597 8453 raft_consensus.cc:348] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.048720 8452 raft_consensus.cc:374] T 3455c9e265e4420698d9478a7c2efee0 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:49.048784 8452 raft_consensus.cc:729] T 3455c9e265e4420698d9478a7c2efee0 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f5ae060d2648b5904ebc7f0b756ab5, State: Initialized, Role: FOLLOWER
I20260929 20:53:49.048815 8453 raft_consensus.cc:374] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:49.048893 8453 raft_consensus.cc:729] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e2bf586cb8c4807b546b3263c73d17e, State: Initialized, Role: FOLLOWER
I20260929 20:53:49.048874 8458 tablet_replica.cc:439] T 3455c9e265e4420698d9478a7c2efee0 P 8a72c22c698049e1923f59f3c839040e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } } }
I20260929 20:53:49.048985 8452 consensus_queue.cc:260] T 3455c9e265e4420698d9478a7c2efee0 P 81f5ae060d2648b5904ebc7f0b756ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.049130 8458 tablet_replica.cc:442] T 3455c9e265e4420698d9478a7c2efee0 P 8a72c22c698049e1923f59f3c839040e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:49.049109 8453 consensus_queue.cc:260] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.049511 8452 ts_tablet_manager.cc:1428] T 3455c9e265e4420698d9478a7c2efee0 P 81f5ae060d2648b5904ebc7f0b756ab5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 20:53:49.049496 8461 tablet_replica.cc:439] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } } }
I20260929 20:53:49.049671 8453 ts_tablet_manager.cc:1428] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:53:49.049440 8459 tablet_replica.cc:439] T 3455c9e265e4420698d9478a7c2efee0 P 81f5ae060d2648b5904ebc7f0b756ab5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } } }
I20260929 20:53:49.049782 8461 tablet_replica.cc:442] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:49.049791 8459 tablet_replica.cc:442] T 3455c9e265e4420698d9478a7c2efee0 P 81f5ae060d2648b5904ebc7f0b756ab5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:49.050508 8451 ts_tablet_manager.cc:1428] T 3455c9e265e4420698d9478a7c2efee0 P 8a72c22c698049e1923f59f3c839040e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260929 20:53:49.202816 8461 raft_consensus.cc:482] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:53:49.203006 8461 raft_consensus.cc:504] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.203632 8461 leader_election.cc:288] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81f5ae060d2648b5904ebc7f0b756ab5 (127.0.97.1:40873), 8a72c22c698049e1923f59f3c839040e (127.0.97.2:40759)
I20260929 20:53:49.204118 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3455c9e265e4420698d9478a7c2efee0" candidate_uuid: "5e2bf586cb8c4807b546b3263c73d17e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" is_pre_election: true
I20260929 20:53:49.204162 8238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3455c9e265e4420698d9478a7c2efee0" candidate_uuid: "5e2bf586cb8c4807b546b3263c73d17e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a72c22c698049e1923f59f3c839040e" is_pre_election: true
I20260929 20:53:49.204320 8099 raft_consensus.cc:2454] T 3455c9e265e4420698d9478a7c2efee0 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e2bf586cb8c4807b546b3263c73d17e in term 0.
I20260929 20:53:49.204366 8238 raft_consensus.cc:2454] T 3455c9e265e4420698d9478a7c2efee0 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e2bf586cb8c4807b546b3263c73d17e in term 0.
I20260929 20:53:49.204721 8308 leader_election.cc:302] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e2bf586cb8c4807b546b3263c73d17e, 81f5ae060d2648b5904ebc7f0b756ab5; no voters:
I20260929 20:53:49.204973 8461 raft_consensus.cc:2781] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:53:49.205072 8461 raft_consensus.cc:482] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:53:49.205174 8461 raft_consensus.cc:3037] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:49.207043 8461 raft_consensus.cc:504] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.207549 8461 leader_election.cc:288] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e [CANDIDATE]: Term 1 election: Requested vote from peers 81f5ae060d2648b5904ebc7f0b756ab5 (127.0.97.1:40873), 8a72c22c698049e1923f59f3c839040e (127.0.97.2:40759)
I20260929 20:53:49.207868 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3455c9e265e4420698d9478a7c2efee0" candidate_uuid: "5e2bf586cb8c4807b546b3263c73d17e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f5ae060d2648b5904ebc7f0b756ab5"
I20260929 20:53:49.207974 8238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3455c9e265e4420698d9478a7c2efee0" candidate_uuid: "5e2bf586cb8c4807b546b3263c73d17e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a72c22c698049e1923f59f3c839040e"
I20260929 20:53:49.208073 8099 raft_consensus.cc:3037] T 3455c9e265e4420698d9478a7c2efee0 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:49.208163 8238 raft_consensus.cc:3037] T 3455c9e265e4420698d9478a7c2efee0 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:49.209940 8099 raft_consensus.cc:2454] T 3455c9e265e4420698d9478a7c2efee0 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e2bf586cb8c4807b546b3263c73d17e in term 1.
I20260929 20:53:49.209956 8238 raft_consensus.cc:2454] T 3455c9e265e4420698d9478a7c2efee0 P 8a72c22c698049e1923f59f3c839040e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e2bf586cb8c4807b546b3263c73d17e in term 1.
I20260929 20:53:49.210270 8308 leader_election.cc:302] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e2bf586cb8c4807b546b3263c73d17e, 81f5ae060d2648b5904ebc7f0b756ab5; no voters:
I20260929 20:53:49.210520 8461 raft_consensus.cc:2781] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:49.210875 8461 raft_consensus.cc:686] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e [term 1 LEADER]: Becoming Leader. State: Replica: 5e2bf586cb8c4807b546b3263c73d17e, State: Running, Role: LEADER
I20260929 20:53:49.211112 8461 consensus_queue.cc:237] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.212610 8494 tablet_replica.cc:439] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e: TxnStatusTablet state changed. Reason: New leader 5e2bf586cb8c4807b546b3263c73d17e. Latest consensus state: current_term: 1 leader_uuid: "5e2bf586cb8c4807b546b3263c73d17e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } } }
I20260929 20:53:49.212961 8494 tablet_replica.cc:442] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:53:49.213485 8496 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 20:53:49.213693 8496 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 20:53:49.214056 7940 catalog_manager.cc:5495] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e2bf586cb8c4807b546b3263c73d17e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "5e2bf586cb8c4807b546b3263c73d17e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } health_report { overall_health: HEALTHY } } }
I20260929 20:53:49.238580 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.233611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.0.97.3:35543)
0929 20:53:49.233768 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:49.233774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:49.233961 (+ 187us) server_negotiation.cc:407] Connection header received
0929 20:53:49.234236 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:49.234242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:49.234324 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:49.234398 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:49.234760 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.235478 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.236829 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.237346 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.237445 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.237957 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:49.238052 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:49.238116 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:49.238292 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 20:53:49.238357 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 20:53:49.238780 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.233653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35543 (local address 127.0.0.1:43814)
0929 20:53:49.233868 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:49.233894 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:49.234029 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:49.234437 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:49.234441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:49.234454 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:53:49.234625 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:49.234633 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.235601 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.235606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:49.236582 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:49.236593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.237497 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.237503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:49.237721 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.237792 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:53:49.238140 (+ 348us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:53:49.238157 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 20:53:49.238261 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 20:53:49.238425 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 20:53:49.242422 8238 raft_consensus.cc:1261] T 3455c9e265e4420698d9478a7c2efee0 P 8a72c22c698049e1923f59f3c839040e [term 1 FOLLOWER]: Refusing update from remote peer 5e2bf586cb8c4807b546b3263c73d17e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:49.242486 8099 raft_consensus.cc:1261] T 3455c9e265e4420698d9478a7c2efee0 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 FOLLOWER]: Refusing update from remote peer 5e2bf586cb8c4807b546b3263c73d17e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:49.243032 8494 consensus_queue.cc:1035] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:49.243304 8461 consensus_queue.cc:1035] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e [LEADER]: Connected to new peer: Peer: permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:49.246380 8458 tablet_replica.cc:439] T 3455c9e265e4420698d9478a7c2efee0 P 8a72c22c698049e1923f59f3c839040e: TxnStatusTablet state changed. Reason: New leader 5e2bf586cb8c4807b546b3263c73d17e. Latest consensus state: current_term: 1 leader_uuid: "5e2bf586cb8c4807b546b3263c73d17e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } } }
I20260929 20:53:49.246691 8458 tablet_replica.cc:442] T 3455c9e265e4420698d9478a7c2efee0 P 8a72c22c698049e1923f59f3c839040e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:49.249445 8500 mvcc.cc:205] Tried to move back new op lower bound from 7334769578971656192 to 7334769578851221504. Current Snapshot: MvccSnapshot[applied={T|T < 7334769578971656192}]
I20260929 20:53:49.250756 8458 tablet_replica.cc:439] T 3455c9e265e4420698d9478a7c2efee0 P 8a72c22c698049e1923f59f3c839040e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e2bf586cb8c4807b546b3263c73d17e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } } }
I20260929 20:53:49.251026 8458 tablet_replica.cc:442] T 3455c9e265e4420698d9478a7c2efee0 P 8a72c22c698049e1923f59f3c839040e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:49.253684 8494 tablet_replica.cc:439] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e2bf586cb8c4807b546b3263c73d17e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } } }
I20260929 20:53:49.254066 8494 tablet_replica.cc:442] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:53:49.260026 8459 tablet_replica.cc:439] T 3455c9e265e4420698d9478a7c2efee0 P 81f5ae060d2648b5904ebc7f0b756ab5: TxnStatusTablet state changed. Reason: New leader 5e2bf586cb8c4807b546b3263c73d17e. Latest consensus state: current_term: 1 leader_uuid: "5e2bf586cb8c4807b546b3263c73d17e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } } }
I20260929 20:53:49.260331 8459 tablet_replica.cc:442] T 3455c9e265e4420698d9478a7c2efee0 P 81f5ae060d2648b5904ebc7f0b756ab5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:49.266587 8461 tablet_replica.cc:439] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e2bf586cb8c4807b546b3263c73d17e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } } }
I20260929 20:53:49.266966 8461 tablet_replica.cc:442] T 3455c9e265e4420698d9478a7c2efee0 P 5e2bf586cb8c4807b546b3263c73d17e: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:53:49.267587 8459 tablet_replica.cc:439] T 3455c9e265e4420698d9478a7c2efee0 P 81f5ae060d2648b5904ebc7f0b756ab5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e2bf586cb8c4807b546b3263c73d17e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } } }
I20260929 20:53:49.267819 8459 tablet_replica.cc:442] T 3455c9e265e4420698d9478a7c2efee0 P 81f5ae060d2648b5904ebc7f0b756ab5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:53:49.277697 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.271099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39966 (local address 127.0.97.2:40759)
0929 20:53:49.271571 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:49.271577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:49.271850 (+ 273us) server_negotiation.cc:407] Connection header received
0929 20:53:49.272054 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:49.272057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:49.272141 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:49.272274 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:49.272756 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.273899 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.275236 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.275878 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.276002 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.276335 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:49.277067 (+ 732us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:49.277208 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:49.277253 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 20:53:49.277333 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":229,"threads_started":1}
I20260929 20:53:49.293639 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.287776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51835 (local address 127.0.97.3:35543)
0929 20:53:49.287984 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:49.287990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:49.288532 (+ 542us) server_negotiation.cc:407] Connection header received
0929 20:53:49.288687 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:49.288690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:49.288752 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:49.288821 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:49.289315 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.290045 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.291985 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.292435 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.292534 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.293003 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:49.293142 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:49.293205 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:49.293332 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 20:53:49.293386 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 20:53:49.293834 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.287767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35543 (local address 127.0.97.2:51835)
0929 20:53:49.288430 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:49.288449 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:49.288583 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:49.288854 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:49.288859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:49.288872 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:53:49.289186 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:49.289197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.290167 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.290173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:49.291861 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:49.291877 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.292592 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.292598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:49.292775 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.292840 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:53:49.293227 (+ 387us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:53:49.293237 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 20:53:49.293322 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 20:53:49.293467 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":366,"threads_started":1}
I20260929 20:53:49.302654 8098 raft_consensus.cc:1261] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 FOLLOWER]: Refusing update from remote peer 8a72c22c698049e1923f59f3c839040e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:49.302917 8377 raft_consensus.cc:1261] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e [term 1 FOLLOWER]: Refusing update from remote peer 8a72c22c698049e1923f59f3c839040e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:49.303591 8458 consensus_queue.cc:1035] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:49.303877 8482 consensus_queue.cc:1035] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [LEADER]: Connected to new peer: Peer: permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:49.336809 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.330861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34089 (local address 127.0.97.2:40759)
0929 20:53:49.331036 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:49.331042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:49.331603 (+ 561us) server_negotiation.cc:407] Connection header received
0929 20:53:49.331746 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:49.331751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:49.331830 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:49.331927 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:49.332646 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.333357 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.334934 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.335587 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.335692 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.336128 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:49.336252 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:49.336333 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:49.336506 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 20:53:49.336583 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:53:49.337308 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.330831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40759 (local address 127.0.97.3:34089)
0929 20:53:49.331508 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:49.331527 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:49.331639 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:49.331947 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:49.331952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:49.331962 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:53:49.332466 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:49.332479 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.333490 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.333494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:49.334801 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:49.334813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.335765 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.335771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:49.335910 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.335966 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:53:49.336391 (+ 425us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:53:49.336402 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:53:49.336476 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 20:53:49.336916 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":331,"threads_started":1}
I20260929 20:53:49.370222 7940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:49.370502 7940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:49.373907 7940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27: Sending DeleteTablet for 3 replicas of tablet 3a39bf5ae6d84f45ac35fa39e6aff769
I20260929 20:53:49.375021 8079 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a39bf5ae6d84f45ac35fa39e6aff769 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:49 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35030
I20260929 20:53:49.375033 8218 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a39bf5ae6d84f45ac35fa39e6aff769 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:49 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39964
I20260929 20:53:49.375438 8357 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a39bf5ae6d84f45ac35fa39e6aff769 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:49 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43798
I20260929 20:53:49.399657 8523 tablet_replica.cc:330] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e: stopping tablet replica
I20260929 20:53:49.400017 8523 raft_consensus.cc:2229] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:49.400487 8523 raft_consensus.cc:2258] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:49.401188 8521 tablet_replica.cc:330] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5: stopping tablet replica
I20260929 20:53:49.402012 8523 ts_tablet_manager.cc:1905] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:49.402351 8521 raft_consensus.cc:2229] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:49.406402 8521 raft_consensus.cc:2258] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:49.407111 8522 tablet_replica.cc:330] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e: stopping tablet replica
I20260929 20:53:49.407850 8521 ts_tablet_manager.cc:1905] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:49.414387 8522 raft_consensus.cc:2229] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:53:49.415040 8522 raft_consensus.cc:2258] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:49.416216 8522 ts_tablet_manager.cc:1905] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:49.420468 8523 ts_tablet_manager.cc:1918] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:53:49.420598 8523 log.cc:1196] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/3a39bf5ae6d84f45ac35fa39e6aff769
I20260929 20:53:49.421000 8523 ts_tablet_manager.cc:1939] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 5e2bf586cb8c4807b546b3263c73d17e: Deleting consensus metadata
I20260929 20:53:49.422037 7927 catalog_manager.cc:4841] TS 5e2bf586cb8c4807b546b3263c73d17e (127.0.97.3:35543): tablet 3a39bf5ae6d84f45ac35fa39e6aff769 (table test-table [id=c153c13c5fe34340b1e9083d0a2e08f1]) successfully deleted
I20260929 20:53:49.422585 8521 ts_tablet_manager.cc:1918] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:53:49.422708 8521 log.cc:1196] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/3a39bf5ae6d84f45ac35fa39e6aff769
I20260929 20:53:49.423103 8521 ts_tablet_manager.cc:1939] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 81f5ae060d2648b5904ebc7f0b756ab5: Deleting consensus metadata
I20260929 20:53:49.424026 7927 catalog_manager.cc:4841] TS 81f5ae060d2648b5904ebc7f0b756ab5 (127.0.97.1:40873): tablet 3a39bf5ae6d84f45ac35fa39e6aff769 (table test-table [id=c153c13c5fe34340b1e9083d0a2e08f1]) successfully deleted
I20260929 20:53:49.427218 8522 ts_tablet_manager.cc:1918] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:53:49.427366 8522 log.cc:1196] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/3a39bf5ae6d84f45ac35fa39e6aff769
I20260929 20:53:49.427706 8522 ts_tablet_manager.cc:1939] T 3a39bf5ae6d84f45ac35fa39e6aff769 P 8a72c22c698049e1923f59f3c839040e: Deleting consensus metadata
I20260929 20:53:49.428635 7928 catalog_manager.cc:4841] TS 8a72c22c698049e1923f59f3c839040e (127.0.97.2:40759): tablet 3a39bf5ae6d84f45ac35fa39e6aff769 (table test-table [id=c153c13c5fe34340b1e9083d0a2e08f1]) successfully deleted
Sep 29 20:53:49 dist-test-slave-m9g1 krb5kdc[7898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:53:49 dist-test-slave-m9g1 krb5kdc[7898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:53:49 dist-test-slave-m9g1 krb5kdc[7898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:53:49.453967 8490 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:49.444082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.0.97.62:46269)
0929 20:53:49.444284 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:49.444290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:49.444644 (+ 354us) server_negotiation.cc:407] Connection header received
0929 20:53:49.444859 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:49.444863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:49.444924 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:49.445006 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:49.446430 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.447167 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.448622 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.449060 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.449178 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.453765 (+ 4587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51408: BlockingRecv error: recv got EOF from 127.0.0.1:51408 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 20:53:49 dist-test-slave-m9g1 krb5kdc[7898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:53:49.470640 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.457319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51422 (local address 127.0.97.62:46269)
0929 20:53:49.457474 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:49.457478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:49.458075 (+ 597us) server_negotiation.cc:407] Connection header received
0929 20:53:49.458236 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:49.458241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:49.458302 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:49.458386 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:49.459415 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.460091 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.461383 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.461807 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.461909 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.465553 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:49.465559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:49.465563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:49.465600 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:49.467952 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:49.468531 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:49.468537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:49.468541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:49.468662 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:49.469013 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:49.469018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:49.469022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:49.469273 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:49.469508 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:49.470068 (+ 560us) server_negotiation.cc:299] Negotiation successful
0929 20:53:49.470308 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:53:49.473701 7940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:53:49.474306 7940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:53:49.480825 8218 tablet_service.cc:1467] Processing CreateTablet for tablet e641cf044f724c27a6a6819f61c42c64 (DEFAULT_TABLE table=test-table [id=cf9665036c4643f29b7596d1569c036d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:49.481284 8218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e641cf044f724c27a6a6819f61c42c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:49.481293 8079 tablet_service.cc:1467] Processing CreateTablet for tablet e641cf044f724c27a6a6819f61c42c64 (DEFAULT_TABLE table=test-table [id=cf9665036c4643f29b7596d1569c036d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:49.481642 8079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e641cf044f724c27a6a6819f61c42c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:49.486590 8357 tablet_service.cc:1467] Processing CreateTablet for tablet e641cf044f724c27a6a6819f61c42c64 (DEFAULT_TABLE table=test-table [id=cf9665036c4643f29b7596d1569c036d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:49.487046 8357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e641cf044f724c27a6a6819f61c42c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:49.488308 8451 tablet_bootstrap.cc:492] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e: Bootstrap starting.
I20260929 20:53:49.489884 8451 tablet_bootstrap.cc:654] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:49.494699 8452 tablet_bootstrap.cc:492] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5: Bootstrap starting.
I20260929 20:53:49.495505 8453 tablet_bootstrap.cc:492] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e: Bootstrap starting.
I20260929 20:53:49.496170 8452 tablet_bootstrap.cc:654] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:49.497064 8453 tablet_bootstrap.cc:654] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:49.499408 8451 tablet_bootstrap.cc:492] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e: No bootstrap required, opened a new log
I20260929 20:53:49.499550 8451 ts_tablet_manager.cc:1397] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260929 20:53:49.500281 8451 raft_consensus.cc:348] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.500559 8451 raft_consensus.cc:374] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:49.500653 8451 raft_consensus.cc:729] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a72c22c698049e1923f59f3c839040e, State: Initialized, Role: FOLLOWER
I20260929 20:53:49.500869 8451 consensus_queue.cc:260] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.501950 8451 ts_tablet_manager.cc:1428] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 20:53:49.512195 8453 tablet_bootstrap.cc:492] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e: No bootstrap required, opened a new log
I20260929 20:53:49.512393 8453 ts_tablet_manager.cc:1397] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.001s
I20260929 20:53:49.513237 8453 raft_consensus.cc:348] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.513510 8453 raft_consensus.cc:374] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:49.513621 8453 raft_consensus.cc:729] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e2bf586cb8c4807b546b3263c73d17e, State: Initialized, Role: FOLLOWER
I20260929 20:53:49.513780 8452 tablet_bootstrap.cc:492] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5: No bootstrap required, opened a new log
I20260929 20:53:49.513978 8452 ts_tablet_manager.cc:1397] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.006s
I20260929 20:53:49.513954 8453 consensus_queue.cc:260] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.514590 8453 ts_tablet_manager.cc:1428] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 20:53:49.514847 8452 raft_consensus.cc:348] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.515158 8452 raft_consensus.cc:374] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:49.515344 8452 raft_consensus.cc:729] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f5ae060d2648b5904ebc7f0b756ab5, State: Initialized, Role: FOLLOWER
I20260929 20:53:49.515655 8452 consensus_queue.cc:260] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.516240 8452 ts_tablet_manager.cc:1428] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 20:53:49.526299 8459 raft_consensus.cc:482] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:53:49.526511 8459 raft_consensus.cc:504] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.528249 8459 leader_election.cc:288] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a72c22c698049e1923f59f3c839040e (127.0.97.2:40759), 5e2bf586cb8c4807b546b3263c73d17e (127.0.97.3:35543)
I20260929 20:53:49.548261 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.529297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42905 (local address 127.0.97.3:35543)
0929 20:53:49.529495 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:49.529501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:49.540472 (+ 10971us) server_negotiation.cc:407] Connection header received
0929 20:53:49.540680 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:49.540686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:49.540760 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:49.540822 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:49.541808 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.542951 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.545370 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.546704 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.546798 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.546910 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:49.547831 (+ 921us) server_negotiation.cc:299] Negotiation successful
0929 20:53:49.547984 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:53:49.548902 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.528935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53159 (local address 127.0.97.2:40759)
0929 20:53:49.529136 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:49.529142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:49.540210 (+ 11068us) server_negotiation.cc:407] Connection header received
0929 20:53:49.540683 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:49.540689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:49.540764 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:49.540830 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:53:49.541772 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.542988 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.546563 (+ 3575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.547790 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.547874 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.547966 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:49.548506 (+ 540us) server_negotiation.cc:299] Negotiation successful
0929 20:53:49.548639 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:53:49.549479 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.528767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35543 (local address 127.0.97.1:42905)
0929 20:53:49.540375 (+ 11608us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:49.540391 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:49.540544 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:49.541018 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:49.541023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:49.541059 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:49.541636 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:49.541646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.543146 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.543150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:49.545165 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:49.545214 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.547433 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.547440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:49.547605 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.547638 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:53:49.548739 (+ 1101us) client_negotiation.cc:240] Negotiation successful
0929 20:53:49.548929 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11486,"mutex_wait_us":64,"spinlock_wait_cycles":1408,"thread_start_us":11009,"threads_started":1}
I20260929 20:53:49.549602 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.528475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40759 (local address 127.0.97.1:53159)
0929 20:53:49.540069 (+ 11594us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:49.540102 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:49.540514 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:49.541018 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:49.541023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:49.541064 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:53:49.541565 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:49.541582 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.544304 (+ 2722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.544310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:49.546380 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:49.546395 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.548151 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.548156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:49.548316 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.548346 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:53:49.549127 (+ 781us) client_negotiation.cc:240] Negotiation successful
0929 20:53:49.549256 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11445,"mutex_wait_us":45,"spinlock_wait_cycles":4864,"thread_start_us":11757,"threads_started":1}
I20260929 20:53:49.550287 8377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e641cf044f724c27a6a6819f61c42c64" candidate_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e2bf586cb8c4807b546b3263c73d17e" is_pre_election: true
I20260929 20:53:49.550297 8238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e641cf044f724c27a6a6819f61c42c64" candidate_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a72c22c698049e1923f59f3c839040e" is_pre_election: true
I20260929 20:53:49.550534 8238 raft_consensus.cc:2454] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81f5ae060d2648b5904ebc7f0b756ab5 in term 0.
I20260929 20:53:49.550534 8377 raft_consensus.cc:2454] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81f5ae060d2648b5904ebc7f0b756ab5 in term 0.
I20260929 20:53:49.551098 8031 leader_election.cc:302] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81f5ae060d2648b5904ebc7f0b756ab5, 8a72c22c698049e1923f59f3c839040e; no voters:
I20260929 20:53:49.551532 8459 raft_consensus.cc:2781] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:53:49.551656 8459 raft_consensus.cc:482] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:53:49.551746 8459 raft_consensus.cc:3037] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:49.553779 8459 raft_consensus.cc:504] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.554508 8459 leader_election.cc:288] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [CANDIDATE]: Term 1 election: Requested vote from peers 8a72c22c698049e1923f59f3c839040e (127.0.97.2:40759), 5e2bf586cb8c4807b546b3263c73d17e (127.0.97.3:35543)
I20260929 20:53:49.555296 8377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e641cf044f724c27a6a6819f61c42c64" candidate_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e2bf586cb8c4807b546b3263c73d17e"
I20260929 20:53:49.555516 8377 raft_consensus.cc:3037] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:49.555639 8238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e641cf044f724c27a6a6819f61c42c64" candidate_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a72c22c698049e1923f59f3c839040e"
I20260929 20:53:49.555820 8238 raft_consensus.cc:3037] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:49.558668 8238 raft_consensus.cc:2454] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81f5ae060d2648b5904ebc7f0b756ab5 in term 1.
I20260929 20:53:49.558668 8377 raft_consensus.cc:2454] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81f5ae060d2648b5904ebc7f0b756ab5 in term 1.
I20260929 20:53:49.559077 8031 leader_election.cc:302] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5e2bf586cb8c4807b546b3263c73d17e, 81f5ae060d2648b5904ebc7f0b756ab5, 8a72c22c698049e1923f59f3c839040e; no voters:
I20260929 20:53:49.559348 8459 raft_consensus.cc:2781] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:49.559799 8459 raft_consensus.cc:686] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 LEADER]: Becoming Leader. State: Replica: 81f5ae060d2648b5904ebc7f0b756ab5, State: Running, Role: LEADER
I20260929 20:53:49.560387 8459 consensus_queue.cc:237] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.563704 7942 catalog_manager.cc:5495] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81f5ae060d2648b5904ebc7f0b756ab5 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } health_report { overall_health: UNKNOWN } } }
I20260929 20:53:49.579455 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.572129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35040 (local address 127.0.97.1:40873)
0929 20:53:49.574602 (+ 2473us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:49.574606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:49.574625 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:53:49.574696 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:49.574699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:49.574766 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:49.574908 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:49.575307 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.576050 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.577314 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.577721 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.577824 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.578249 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:49.578812 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:49.578937 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:49.579038 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 20:53:49.579111 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2368,"thread_start_us":2245,"threads_started":1}
I20260929 20:53:49.583087 8238 raft_consensus.cc:1261] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e [term 1 FOLLOWER]: Refusing update from remote peer 81f5ae060d2648b5904ebc7f0b756ab5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:49.583211 8377 raft_consensus.cc:1261] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e [term 1 FOLLOWER]: Refusing update from remote peer 81f5ae060d2648b5904ebc7f0b756ab5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:49.583870 8541 consensus_queue.cc:1035] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:49.584468 8459 consensus_queue.cc:1035] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:49.628356 7940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:49.628608 7940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:49.632186 7940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27: Sending DeleteTablet for 3 replicas of tablet e641cf044f724c27a6a6819f61c42c64
I20260929 20:53:49.633528 8079 tablet_service.cc:1514] Processing DeleteTablet for tablet e641cf044f724c27a6a6819f61c42c64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:49 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35030
I20260929 20:53:49.633848 8357 tablet_service.cc:1514] Processing DeleteTablet for tablet e641cf044f724c27a6a6819f61c42c64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:49 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43798
I20260929 20:53:49.634104 8521 tablet_replica.cc:330] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5: stopping tablet replica
I20260929 20:53:49.634305 8521 raft_consensus.cc:2229] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:53:49.634796 8521 raft_consensus.cc:2258] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:49.635063 8523 tablet_replica.cc:330] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e: stopping tablet replica
I20260929 20:53:49.635239 8523 raft_consensus.cc:2229] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:49.635442 8523 raft_consensus.cc:2258] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:49.636487 8218 tablet_service.cc:1514] Processing DeleteTablet for tablet e641cf044f724c27a6a6819f61c42c64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:49 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39964
I20260929 20:53:49.636760 8522 tablet_replica.cc:330] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e: stopping tablet replica
I20260929 20:53:49.636951 8522 raft_consensus.cc:2229] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:49.637163 8522 raft_consensus.cc:2258] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:49.638957 8522 ts_tablet_manager.cc:1905] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:49.642860 8521 ts_tablet_manager.cc:1905] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:49.645488 7940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:53:49.646438 7940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:53:49.658911 8218 tablet_service.cc:1467] Processing CreateTablet for tablet 2963af0daa334eb69297e20afe9eca62 (DEFAULT_TABLE table=test-table [id=3982e77cb8ed451a85f14e4d9d87b06a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:49.659387 8218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2963af0daa334eb69297e20afe9eca62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:49.663821 8357 tablet_service.cc:1467] Processing CreateTablet for tablet 2963af0daa334eb69297e20afe9eca62 (DEFAULT_TABLE table=test-table [id=3982e77cb8ed451a85f14e4d9d87b06a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:49.664310 8357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2963af0daa334eb69297e20afe9eca62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:49.663343 8079 tablet_service.cc:1467] Processing CreateTablet for tablet 2963af0daa334eb69297e20afe9eca62 (DEFAULT_TABLE table=test-table [id=3982e77cb8ed451a85f14e4d9d87b06a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:53:49.667052 8079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2963af0daa334eb69297e20afe9eca62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:49.677884 8451 tablet_bootstrap.cc:492] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e: Bootstrap starting.
I20260929 20:53:49.679474 8451 tablet_bootstrap.cc:654] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:49.682782 8452 tablet_bootstrap.cc:492] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5: Bootstrap starting.
I20260929 20:53:49.684296 8452 tablet_bootstrap.cc:654] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:49.685294 8521 ts_tablet_manager.cc:1918] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:49.686646 8521 log.cc:1196] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/e641cf044f724c27a6a6819f61c42c64
I20260929 20:53:49.686931 8523 ts_tablet_manager.cc:1905] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:49.687260 8522 ts_tablet_manager.cc:1918] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:49.687584 8521 ts_tablet_manager.cc:1939] T e641cf044f724c27a6a6819f61c42c64 P 81f5ae060d2648b5904ebc7f0b756ab5: Deleting consensus metadata
I20260929 20:53:49.687749 8522 log.cc:1196] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/e641cf044f724c27a6a6819f61c42c64
I20260929 20:53:49.688345 8522 ts_tablet_manager.cc:1939] T e641cf044f724c27a6a6819f61c42c64 P 8a72c22c698049e1923f59f3c839040e: Deleting consensus metadata
I20260929 20:53:49.692878 8453 tablet_bootstrap.cc:492] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e: Bootstrap starting.
I20260929 20:53:49.694618 8453 tablet_bootstrap.cc:654] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:49.695586 8523 ts_tablet_manager.cc:1918] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:53:49.695725 8523 log.cc:1196] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/e641cf044f724c27a6a6819f61c42c64
I20260929 20:53:49.697248 8523 ts_tablet_manager.cc:1939] T e641cf044f724c27a6a6819f61c42c64 P 5e2bf586cb8c4807b546b3263c73d17e: Deleting consensus metadata
I20260929 20:53:49.698280 7927 catalog_manager.cc:4841] TS 5e2bf586cb8c4807b546b3263c73d17e (127.0.97.3:35543): tablet e641cf044f724c27a6a6819f61c42c64 (table test-table [id=cf9665036c4643f29b7596d1569c036d]) successfully deleted
I20260929 20:53:49.703244 8452 tablet_bootstrap.cc:492] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5: No bootstrap required, opened a new log
I20260929 20:53:49.703305 7927 catalog_manager.cc:4841] TS 81f5ae060d2648b5904ebc7f0b756ab5 (127.0.97.1:40873): tablet e641cf044f724c27a6a6819f61c42c64 (table test-table [id=cf9665036c4643f29b7596d1569c036d]) successfully deleted
I20260929 20:53:49.703408 8452 ts_tablet_manager.cc:1397] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.001s
I20260929 20:53:49.704139 8452 raft_consensus.cc:348] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.704380 8452 raft_consensus.cc:374] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:49.704453 8452 raft_consensus.cc:729] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f5ae060d2648b5904ebc7f0b756ab5, State: Initialized, Role: FOLLOWER
I20260929 20:53:49.704464 7928 catalog_manager.cc:4841] TS 8a72c22c698049e1923f59f3c839040e (127.0.97.2:40759): tablet e641cf044f724c27a6a6819f61c42c64 (table test-table [id=cf9665036c4643f29b7596d1569c036d]) successfully deleted
I20260929 20:53:49.704675 8452 consensus_queue.cc:260] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.705170 8452 ts_tablet_manager.cc:1428] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 20:53:49.696346 8451 tablet_bootstrap.cc:492] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e: No bootstrap required, opened a new log
I20260929 20:53:49.707701 8451 ts_tablet_manager.cc:1397] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.011s
I20260929 20:53:49.708459 8451 raft_consensus.cc:348] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.708699 8451 raft_consensus.cc:374] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:49.708786 8451 raft_consensus.cc:729] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a72c22c698049e1923f59f3c839040e, State: Initialized, Role: FOLLOWER
I20260929 20:53:49.709049 8451 consensus_queue.cc:260] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.711232 8453 tablet_bootstrap.cc:492] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e: No bootstrap required, opened a new log
I20260929 20:53:49.711397 8453 ts_tablet_manager.cc:1397] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20260929 20:53:49.711553 8451 ts_tablet_manager.cc:1428] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260929 20:53:49.712190 8453 raft_consensus.cc:348] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.712440 8453 raft_consensus.cc:374] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:49.712532 8453 raft_consensus.cc:729] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e2bf586cb8c4807b546b3263c73d17e, State: Initialized, Role: FOLLOWER
I20260929 20:53:49.712773 8453 consensus_queue.cc:260] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.713279 8453 ts_tablet_manager.cc:1428] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:53:49.774964 8547 raft_consensus.cc:482] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:53:49.775194 8547 raft_consensus.cc:504] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.776017 8547 leader_election.cc:288] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a72c22c698049e1923f59f3c839040e (127.0.97.2:40759), 5e2bf586cb8c4807b546b3263c73d17e (127.0.97.3:35543)
I20260929 20:53:49.776633 8377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2963af0daa334eb69297e20afe9eca62" candidate_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e2bf586cb8c4807b546b3263c73d17e" is_pre_election: true
I20260929 20:53:49.776621 8238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2963af0daa334eb69297e20afe9eca62" candidate_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a72c22c698049e1923f59f3c839040e" is_pre_election: true
I20260929 20:53:49.776866 8377 raft_consensus.cc:2454] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81f5ae060d2648b5904ebc7f0b756ab5 in term 0.
I20260929 20:53:49.776922 8238 raft_consensus.cc:2454] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81f5ae060d2648b5904ebc7f0b756ab5 in term 0.
I20260929 20:53:49.777351 8030 leader_election.cc:302] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e2bf586cb8c4807b546b3263c73d17e, 81f5ae060d2648b5904ebc7f0b756ab5; no voters:
I20260929 20:53:49.777657 8547 raft_consensus.cc:2781] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:53:49.777794 8547 raft_consensus.cc:482] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:53:49.777895 8547 raft_consensus.cc:3037] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:49.780061 8547 raft_consensus.cc:504] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.780994 8377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2963af0daa334eb69297e20afe9eca62" candidate_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e2bf586cb8c4807b546b3263c73d17e"
I20260929 20:53:49.781000 8238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2963af0daa334eb69297e20afe9eca62" candidate_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a72c22c698049e1923f59f3c839040e"
I20260929 20:53:49.781184 8377 raft_consensus.cc:3037] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:49.781194 8238 raft_consensus.cc:3037] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:49.783933 8377 raft_consensus.cc:2454] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81f5ae060d2648b5904ebc7f0b756ab5 in term 1.
I20260929 20:53:49.784349 8238 raft_consensus.cc:2454] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81f5ae060d2648b5904ebc7f0b756ab5 in term 1.
I20260929 20:53:49.784678 8030 leader_election.cc:302] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e2bf586cb8c4807b546b3263c73d17e, 81f5ae060d2648b5904ebc7f0b756ab5; no voters:
I20260929 20:53:49.784972 8459 raft_consensus.cc:2781] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:49.785099 8459 raft_consensus.cc:686] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 LEADER]: Becoming Leader. State: Replica: 81f5ae060d2648b5904ebc7f0b756ab5, State: Running, Role: LEADER
I20260929 20:53:49.785351 8459 consensus_queue.cc:237] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } }
I20260929 20:53:49.787807 8547 leader_election.cc:288] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [CANDIDATE]: Term 1 election: Requested vote from peers 8a72c22c698049e1923f59f3c839040e (127.0.97.2:40759), 5e2bf586cb8c4807b546b3263c73d17e (127.0.97.3:35543)
I20260929 20:53:49.789068 7942 catalog_manager.cc:5495] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81f5ae060d2648b5904ebc7f0b756ab5 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f5ae060d2648b5904ebc7f0b756ab5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 } health_report { overall_health: UNKNOWN } } }
I20260929 20:53:49.843233 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.832263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57337 (local address 127.0.97.3:35543)
0929 20:53:49.832446 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:49.832452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:49.838539 (+ 6087us) server_negotiation.cc:407] Connection header received
0929 20:53:49.838728 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:49.838732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:49.838807 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:49.838881 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:49.839395 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.840054 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.841502 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.841984 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.842103 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.842570 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:49.842669 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:49.842739 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:49.842869 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 20:53:49.842952 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:53:49.843333 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.832270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35543 (local address 127.0.97.1:57337)
0929 20:53:49.838430 (+ 6160us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:49.838450 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:49.838598 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:49.838914 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:49.838919 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:49.838934 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:53:49.839257 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:49.839267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.840176 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.840181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:49.841352 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:49.841365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.842161 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.842167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:49.842335 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.842390 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:53:49.842759 (+ 369us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:53:49.842774 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 20:53:49.842843 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 20:53:49.842978 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5974,"thread_start_us":5818,"threads_started":1}
I20260929 20:53:49.853134 8238 raft_consensus.cc:1261] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e [term 1 FOLLOWER]: Refusing update from remote peer 81f5ae060d2648b5904ebc7f0b756ab5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:49.853672 8377 raft_consensus.cc:1261] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e [term 1 FOLLOWER]: Refusing update from remote peer 81f5ae060d2648b5904ebc7f0b756ab5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:53:49.853699 8459 consensus_queue.cc:1035] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a72c22c698049e1923f59f3c839040e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:49.854557 8459 consensus_queue.cc:1035] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e2bf586cb8c4807b546b3263c73d17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:53:49.903707 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.895927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39559 (local address 127.0.97.1:40873)
0929 20:53:49.896079 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:49.896085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:49.898688 (+ 2603us) server_negotiation.cc:407] Connection header received
0929 20:53:49.898843 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:49.898848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:49.898920 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:49.898990 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:53:49.899461 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.900528 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.902027 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.902512 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.902617 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.902971 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:53:49.903070 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:53:49.903157 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:49.903293 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 20:53:49.903371 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:53:49.904440 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.895611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40873 (local address 127.0.97.3:39559)
0929 20:53:49.898592 (+ 2981us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:49.898611 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:49.898727 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:49.899018 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:49.899022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:49.899034 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:53:49.899285 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:49.899298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.900699 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.900704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:49.901867 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:49.901879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.902639 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.902644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:49.902793 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.902829 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:53:49.903158 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:53:49.903170 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 20:53:49.903952 (+ 782us) client_negotiation.cc:240] Negotiation successful
0929 20:53:49.904090 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2859,"thread_start_us":2731,"threads_started":1}
I20260929 20:53:49.956202 7942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:49.956426 7942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:53:49.959069 7942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d12677949284c699c61ae92f95a6a27: Sending DeleteTablet for 3 replicas of tablet 2963af0daa334eb69297e20afe9eca62
I20260929 20:53:49.959862 8079 tablet_service.cc:1514] Processing DeleteTablet for tablet 2963af0daa334eb69297e20afe9eca62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:49 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35030
I20260929 20:53:49.959949 8218 tablet_service.cc:1514] Processing DeleteTablet for tablet 2963af0daa334eb69297e20afe9eca62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:49 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39964
I20260929 20:53:49.960222 8521 tablet_replica.cc:330] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5: stopping tablet replica
I20260929 20:53:49.960253 8522 tablet_replica.cc:330] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e: stopping tablet replica
I20260929 20:53:49.960410 8521 raft_consensus.cc:2229] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:53:49.960415 8522 raft_consensus.cc:2229] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:49.960599 8522 raft_consensus.cc:2258] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:49.960712 8521 raft_consensus.cc:2258] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:49.961227 8357 tablet_service.cc:1514] Processing DeleteTablet for tablet 2963af0daa334eb69297e20afe9eca62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:53:49 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43798
I20260929 20:53:49.961494 8523 tablet_replica.cc:330] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e: stopping tablet replica
I20260929 20:53:49.961642 8523 raft_consensus.cc:2229] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:53:49.961798 8521 ts_tablet_manager.cc:1905] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:49.961808 8523 raft_consensus.cc:2258] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:53:49.962709 8523 ts_tablet_manager.cc:1905] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:49.965943 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 8014
I20260929 20:53:49.969131 8523 ts_tablet_manager.cc:1918] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:53:49.969358 8523 log.cc:1196] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/2963af0daa334eb69297e20afe9eca62
I20260929 20:53:49.969828 8523 ts_tablet_manager.cc:1939] T 2963af0daa334eb69297e20afe9eca62 P 5e2bf586cb8c4807b546b3263c73d17e: Deleting consensus metadata
I20260929 20:53:49.970958 7927 catalog_manager.cc:4841] TS 5e2bf586cb8c4807b546b3263c73d17e (127.0.97.3:35543): tablet 2963af0daa334eb69297e20afe9eca62 (table test-table [id=3982e77cb8ed451a85f14e4d9d87b06a]) successfully deleted
I20260929 20:53:49.972164 8521 ts_tablet_manager.cc:1918] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:53:49.972280 8521 log.cc:1196] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/2963af0daa334eb69297e20afe9eca62
I20260929 20:53:49.972625 8521 ts_tablet_manager.cc:1939] T 2963af0daa334eb69297e20afe9eca62 P 81f5ae060d2648b5904ebc7f0b756ab5: Deleting consensus metadata
I20260929 20:53:49.973559 7927 catalog_manager.cc:4841] TS 81f5ae060d2648b5904ebc7f0b756ab5 (127.0.97.1:40873): tablet 2963af0daa334eb69297e20afe9eca62 (table test-table [id=3982e77cb8ed451a85f14e4d9d87b06a]) successfully deleted
I20260929 20:53:49.975282 8522 ts_tablet_manager.cc:1905] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:53:49.982699 8522 ts_tablet_manager.cc:1918] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:53:49.982897 8522 log.cc:1196] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/2963af0daa334eb69297e20afe9eca62
I20260929 20:53:49.983340 8522 ts_tablet_manager.cc:1939] T 2963af0daa334eb69297e20afe9eca62 P 8a72c22c698049e1923f59f3c839040e: Deleting consensus metadata
I20260929 20:53:49.984525 7928 catalog_manager.cc:4841] TS 8a72c22c698049e1923f59f3c839040e (127.0.97.2:40759): tablet 2963af0daa334eb69297e20afe9eca62 (table test-table [id=3982e77cb8ed451a85f14e4d9d87b06a]) successfully deleted
Sep 29 20:53:49 dist-test-slave-m9g1 krb5kdc[7898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260929 20:53:49.997045 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:49.967022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56456 (local address 127.0.97.1:40873)
0929 20:53:49.967177 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:49.967182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:49.982349 (+ 15167us) server_negotiation.cc:407] Connection header received
0929 20:53:49.982862 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:49.982868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:49.983000 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:49.983086 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:49.984922 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.985675 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.986982 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:49.987424 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:49.987532 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:49.991165 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:49.991175 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:49.991189 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:49.991224 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:49.994350 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:49.994970 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:49.994983 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:49.994987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:49.995126 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:49.995486 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:49.995491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:49.995494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:49.995859 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:49.996148 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:49.996451 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 20:53:49.996674 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 20:53:49.997669 8144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:50.122748 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 8014
I20260929 20:53:50.139007 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 8153
Sep 29 20:53:50 dist-test-slave-m9g1 krb5kdc[7898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260929 20:53:50.154315 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:50.139690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43454 (local address 127.0.97.2:40759)
0929 20:53:50.140137 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:50.140142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:50.140157 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:53:50.140212 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:50.140215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:50.140271 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:50.140376 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:50.141448 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:50.142401 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:50.143629 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:50.144247 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:50.144376 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:50.148301 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:50.148319 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:50.148341 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:50.148381 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:50.151759 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:50.152354 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:50.152363 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:50.152367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:50.152527 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:50.152811 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:50.152816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:50.152820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:50.153227 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:50.153492 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:50.153709 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 20:53:50.153926 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":202,"threads_started":1}
I20260929 20:53:50.154994 8283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:50.272102 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 8153
I20260929 20:53:50.289439 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 8292
Sep 29 20:53:50 dist-test-slave-m9g1 krb5kdc[7898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260929 20:53:50.302433 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:50.290240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55026 (local address 127.0.97.3:35543)
0929 20:53:50.290404 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:50.290408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:50.290424 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:53:50.290497 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:50.290501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:50.290556 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:50.290632 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:50.291734 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:50.292399 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:50.293558 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:50.294040 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:50.294137 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:50.297878 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:50.297892 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:50.297909 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:50.297939 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:50.300364 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:50.300922 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:50.300929 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:50.300932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:50.301049 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:50.301305 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:50.301307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:50.301319 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:50.301612 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:50.301821 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:50.302014 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 20:53:50.302159 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:53:50.302985 8422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:50.439481 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 8292
I20260929 20:53:50.459545 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 7911
I20260929 20:53:50.460664 7983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:50.571229 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 7911
2026-09-29T20:53:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5564 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:53:50 dist-test-slave-m9g1 krb5kdc[8571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:53:50 dist-test-slave-m9g1 krb5kdc[8571](info): set up 2 sockets
Sep 29 20:53:50 dist-test-slave-m9g1 krb5kdc[8571](info): commencing operation
krb5kdc: starting...
W20260929 20:53:51.578809 388 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:53:51 dist-test-slave-m9g1 krb5kdc[8571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:53:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:53:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:53:51.771567 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41465
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:52.074295 8584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:52.074616 8584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:52.074750 8584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:52.081761 8584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:53:52.081878 8584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:52.081956 8584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:52.082028 8584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:53:52.082090 8584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:53:52.091789 8584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:52.093019 8584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:52.094322 8584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:52.104848 8590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:52.104912 8592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:52.104887 8589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:52.105762 8584 server_base.cc:1034] running on GCE node
I20260929 20:53:52.106525 8584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:52.107796 8584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:52.108987 8584 hybrid_clock.cc:648] HybridClock initialized: now 1790715232108932 us; error 44 us; skew 500 ppm
I20260929 20:53:52.109275 8584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T20:53:52Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1602 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:53:52 dist-test-slave-m9g1 krb5kdc[8598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:53:52 dist-test-slave-m9g1 krb5kdc[8598](info): set up 2 sockets
Sep 29 20:53:52 dist-test-slave-m9g1 krb5kdc[8598](info): commencing operation
krb5kdc: starting...
W20260929 20:53:53.178792 388 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:53:53 dist-test-slave-m9g1 krb5kdc[8598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:53:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:53:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:53:53.371433 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36703
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:53.672302 8611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:53.672557 8611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:53.672650 8611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:53.679433 8611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:53:53.679520 8611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:53.679570 8611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:53.679628 8611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:53:53.679669 8611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:53:53.689265 8611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:53.690421 8611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:53.691571 8611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:53.702214 8616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:53.702268 8619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:53.702293 8617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:53.703235 8611 server_base.cc:1034] running on GCE node
I20260929 20:53:53.703945 8611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:53.705286 8611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:53.706472 8611 hybrid_clock.cc:648] HybridClock initialized: now 1790715233706422 us; error 43 us; skew 500 ppm
I20260929 20:53:53.706789 8611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:53 dist-test-slave-m9g1 krb5kdc[8598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:53.709568 8611 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:53:53.710970 8611 webserver.cc:460] Webserver started at http://127.0.97.62:38817/ using document root <none> and password file <none>
I20260929 20:53:53.711607 8611 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:53.711715 8611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:53.711994 8611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:53.714864 8611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "9cbd9e7796184921b3918c65be371e1c"
format_stamp: "Formatted at 2026-09-29 20:53:53 on dist-test-slave-m9g1"
server_key: "06839fccb30a3f4a32078f8149a47c30"
server_key_iv: "36e8e6712de4ad53050475c181a7629e"
server_key_version: "encryptionkey@0"
I20260929 20:53:53.715588 8611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "9cbd9e7796184921b3918c65be371e1c"
format_stamp: "Formatted at 2026-09-29 20:53:53 on dist-test-slave-m9g1"
server_key: "06839fccb30a3f4a32078f8149a47c30"
server_key_iv: "36e8e6712de4ad53050475c181a7629e"
server_key_version: "encryptionkey@0"
I20260929 20:53:53.720363 8611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:53:53.723623 8626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:53.724872 8611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:53:53.725018 8611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "9cbd9e7796184921b3918c65be371e1c"
format_stamp: "Formatted at 2026-09-29 20:53:53 on dist-test-slave-m9g1"
server_key: "06839fccb30a3f4a32078f8149a47c30"
server_key_iv: "36e8e6712de4ad53050475c181a7629e"
server_key_version: "encryptionkey@0"
I20260929 20:53:53.725201 8611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:53.744203 8611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:53.747848 8611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:53.748102 8611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:53.763399 8611 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36703
I20260929 20:53:53.763422 8677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36703 every 8 connection(s)
I20260929 20:53:53.764647 8611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:53:53.768062 8678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:53:53.768464 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 8611
I20260929 20:53:53.768658 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:53:53.769264 388 external_mini_cluster.cc:1428] Setting key 2ca9b5e699201560182da5ab638e561a
I20260929 20:53:53.781487 8678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c: Bootstrap starting.
Sep 29 20:53:53 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:53.784674 8678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c: Neither blocks nor log segments found. Creating new log.
I20260929 20:53:53.785998 8678 log.cc:824] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:53.789144 8678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c: No bootstrap required, opened a new log
I20260929 20:53:53.792068 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:53.771105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41048 (local address 127.0.97.62:36703)
0929 20:53:53.771878 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:53.771893 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:53.771948 (+ 55us) server_negotiation.cc:407] Connection header received
0929 20:53:53.773968 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:53.773996 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:53.774522 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:53.774976 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:53.776168 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:53.778056 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:53.779332 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:53.780376 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:53.780559 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:53.784211 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:53.784222 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:53.784241 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:53.784289 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:53.787676 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:53.788354 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:53.788363 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:53.788372 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:53.788564 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:53.788905 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:53.788910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:53.788913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:53.789606 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:53.789882 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:53.790259 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 20:53:53.790705 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":302,"threads_started":1}
I20260929 20:53:53.797482 8678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbd9e7796184921b3918c65be371e1c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36703 } }
I20260929 20:53:53.797816 8678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:53:53.797888 8678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd9e7796184921b3918c65be371e1c, State: Initialized, Role: FOLLOWER
I20260929 20:53:53.798396 8678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbd9e7796184921b3918c65be371e1c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36703 } }
I20260929 20:53:53.798594 8678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:53:53.798689 8678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:53:53.798797 8678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:53:53.800990 8678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbd9e7796184921b3918c65be371e1c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36703 } }
I20260929 20:53:53.801482 8678 leader_election.cc:302] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cbd9e7796184921b3918c65be371e1c; no voters:
I20260929 20:53:53.801966 8678 leader_election.cc:288] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:53:53.802085 8683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:53:53.802701 8683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [term 1 LEADER]: Becoming Leader. State: Replica: 9cbd9e7796184921b3918c65be371e1c, State: Running, Role: LEADER
I20260929 20:53:53.803185 8683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbd9e7796184921b3918c65be371e1c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36703 } }
I20260929 20:53:53.803893 8678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:53:53.805773 8684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cbd9e7796184921b3918c65be371e1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbd9e7796184921b3918c65be371e1c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36703 } } }
I20260929 20:53:53.806146 8684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:53.806046 8685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cbd9e7796184921b3918c65be371e1c. Latest consensus state: current_term: 1 leader_uuid: "9cbd9e7796184921b3918c65be371e1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbd9e7796184921b3918c65be371e1c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36703 } } }
I20260929 20:53:53.806666 8685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:53.808529 8691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:53:53.813601 8691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:53:53.822000 8691 catalog_manager.cc:1347] Generated new cluster ID: aaa97be01e49486daf4ff9ba05eccd91
I20260929 20:53:53.822106 8691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:53:53.837312 8691 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:53:53.838606 8691 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:53:53.848768 8691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c: Generated new TSK 0
I20260929 20:53:53.849790 8691 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:53:53.909117 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36703
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:54.220773 8704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:54.221046 8704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:54.221177 8704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:54.228772 8704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:54.228885 8704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:54.229046 8704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:53:54.238691 8704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:54.239917 8704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:54.241153 8704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:54.253190 8710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:54.253160 8712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:54.253242 8709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:54.253850 8704 server_base.cc:1034] running on GCE node
I20260929 20:53:54.254427 8704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:54.255250 8704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:54.256438 8704 hybrid_clock.cc:648] HybridClock initialized: now 1790715234256393 us; error 34 us; skew 500 ppm
I20260929 20:53:54.256711 8704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:54 dist-test-slave-m9g1 krb5kdc[8598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:54.259603 8704 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:53:54.260896 8704 webserver.cc:460] Webserver started at http://127.0.97.1:41929/ using document root <none> and password file <none>
I20260929 20:53:54.261505 8704 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:54.261615 8704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:54.261885 8704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:54.264637 8704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "04b53758e3e043128c70bfec0ed2e746"
format_stamp: "Formatted at 2026-09-29 20:53:54 on dist-test-slave-m9g1"
server_key: "f2a60bdef02e326df74faf0487f35e5d"
server_key_iv: "1a038c0a9366e6002589ca3144dae0f3"
server_key_version: "encryptionkey@0"
I20260929 20:53:54.265367 8704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "04b53758e3e043128c70bfec0ed2e746"
format_stamp: "Formatted at 2026-09-29 20:53:54 on dist-test-slave-m9g1"
server_key: "f2a60bdef02e326df74faf0487f35e5d"
server_key_iv: "1a038c0a9366e6002589ca3144dae0f3"
server_key_version: "encryptionkey@0"
I20260929 20:53:54.270109 8704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:53:54.273347 8719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:54.274653 8704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 20:53:54.274796 8704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "04b53758e3e043128c70bfec0ed2e746"
format_stamp: "Formatted at 2026-09-29 20:53:54 on dist-test-slave-m9g1"
server_key: "f2a60bdef02e326df74faf0487f35e5d"
server_key_iv: "1a038c0a9366e6002589ca3144dae0f3"
server_key_version: "encryptionkey@0"
I20260929 20:53:54.274983 8704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:54.297340 8704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:54.301482 8704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:54.301733 8704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:54.302680 8704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:54.303933 8704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:54.304010 8704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:54.304122 8704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:54.304171 8704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:54.332942 8704 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35323
I20260929 20:53:54.332986 8831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35323 every 8 connection(s)
I20260929 20:53:54.334151 8704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:53:54.340045 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 8704
I20260929 20:53:54.340207 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:53:54.340780 388 external_mini_cluster.cc:1428] Setting key d88c21f4da041847dd65852eadd97477
Sep 29 20:53:54 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:54.353098 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:54.336740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37293 (local address 127.0.97.62:36703)
0929 20:53:54.337174 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:54.337179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:54.338225 (+ 1046us) server_negotiation.cc:407] Connection header received
0929 20:53:54.340041 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:54.340047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:54.340131 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:54.340240 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:54.341722 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:54.342500 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:54.343998 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:54.344383 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:54.344490 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:54.347729 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:54.347735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:54.347738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:54.347771 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:54.350292 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:54.350957 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:54.350963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:54.350966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:54.351076 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:54.351476 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:54.351482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:54.351485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:54.351744 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:54.351869 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:54.352517 (+ 648us) server_negotiation.cc:299] Negotiation successful
0929 20:53:54.352733 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":216,"threads_started":1}
I20260929 20:53:54.354059 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:54.337327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.1:37293)
0929 20:53:54.338044 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:54.338086 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:54.339780 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:54.340478 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:54.340486 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:54.341010 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:54.341562 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:54.341583 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:54.342636 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:54.342640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:54.343869 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:54.343881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:54.344585 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:54.344593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:54.344800 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:54.345412 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:54.345435 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:54.347493 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:54.350411 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:54.350421 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:54.350437 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:54.350840 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:54.351178 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:54.351183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:54.351187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:54.351375 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:54.351870 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:54.351877 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:54.352189 (+ 312us) client_negotiation.cc:769] Sending connection context
0929 20:53:54.352429 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 20:53:54.352964 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":287,"threads_started":1}
I20260929 20:53:54.355540 8832 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36703
I20260929 20:53:54.355811 8832 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:54.356468 8832 heartbeater.cc:511] Master 127.0.97.62:36703 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:53:54.358644 8643 ts_manager.cc:194] Registered new tserver with Master: 04b53758e3e043128c70bfec0ed2e746 (127.0.97.1:35323)
I20260929 20:53:54.360056 8643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:53:54.380945 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36703
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:54.685667 8838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:54.685940 8838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:54.686044 8838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:54.693163 8838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:54.693274 8838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:54.693392 8838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:53:54.703255 8838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:54.704543 8838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:54.705825 8838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:54.717952 8844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:54.717993 8846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:54.717916 8843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:54.718737 8838 server_base.cc:1034] running on GCE node
I20260929 20:53:54.719283 8838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:54.720135 8838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:54.721328 8838 hybrid_clock.cc:648] HybridClock initialized: now 1790715234721279 us; error 36 us; skew 500 ppm
I20260929 20:53:54.721608 8838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:54 dist-test-slave-m9g1 krb5kdc[8598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:54.724447 8838 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:53:54.725697 8838 webserver.cc:460] Webserver started at http://127.0.97.2:42859/ using document root <none> and password file <none>
I20260929 20:53:54.726416 8838 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:54.726531 8838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:54.726819 8838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:54.729563 8838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "066280fcd5c4400b9e2cc3d51d38e5c6"
format_stamp: "Formatted at 2026-09-29 20:53:54 on dist-test-slave-m9g1"
server_key: "d4fd1af990f0a84816ff652587a9d645"
server_key_iv: "24ef0d03e30f89bf1792e32ac99fd248"
server_key_version: "encryptionkey@0"
I20260929 20:53:54.730324 8838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "066280fcd5c4400b9e2cc3d51d38e5c6"
format_stamp: "Formatted at 2026-09-29 20:53:54 on dist-test-slave-m9g1"
server_key: "d4fd1af990f0a84816ff652587a9d645"
server_key_iv: "24ef0d03e30f89bf1792e32ac99fd248"
server_key_version: "encryptionkey@0"
I20260929 20:53:54.734894 8838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:53:54.738022 8853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:54.739293 8838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:53:54.739434 8838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "066280fcd5c4400b9e2cc3d51d38e5c6"
format_stamp: "Formatted at 2026-09-29 20:53:54 on dist-test-slave-m9g1"
server_key: "d4fd1af990f0a84816ff652587a9d645"
server_key_iv: "24ef0d03e30f89bf1792e32ac99fd248"
server_key_version: "encryptionkey@0"
I20260929 20:53:54.739603 8838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:54.753389 8838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:54.756804 8838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:54.757052 8838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:54.757858 8838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:54.759188 8838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:54.759262 8838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:54.759366 8838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:54.759419 8838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:54.792045 8838 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38105
I20260929 20:53:54.792076 8965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38105 every 8 connection(s)
I20260929 20:53:54.793239 8838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:53:54.798377 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 8838
I20260929 20:53:54.798528 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:53:54.798936 388 external_mini_cluster.cc:1428] Setting key fed730d3bada82623cd54f0fad83fc6f
Sep 29 20:53:54 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:54.811815 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:54.795574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35909 (local address 127.0.97.62:36703)
0929 20:53:54.795738 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:54.795742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:54.796876 (+ 1134us) server_negotiation.cc:407] Connection header received
0929 20:53:54.798649 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:54.798654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:54.798735 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:54.798819 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:54.800583 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:54.801328 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:54.802887 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:54.803273 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:54.803379 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:54.806679 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:54.806683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:54.806687 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:54.806716 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:54.809033 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:54.809700 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:54.809704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:54.809707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:54.809833 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:54.810250 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:54.810254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:54.810257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:54.810507 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:54.810628 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:54.811279 (+ 651us) server_negotiation.cc:299] Negotiation successful
0929 20:53:54.811505 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:53:54.812690 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:54.796046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.2:35909)
0929 20:53:54.796729 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:54.796765 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:54.798381 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:54.799068 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:54.799080 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:54.799684 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:54.800389 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:54.800413 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:54.801483 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:54.801487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:54.802760 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:54.802771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:54.803477 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:54.803485 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:54.803708 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:54.804291 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:54.804321 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:54.806486 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:54.809172 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:54.809181 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:54.809198 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:54.809584 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:54.809931 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:54.809936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:54.809941 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:54.810127 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:54.810623 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:54.810630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:54.810937 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 20:53:54.811163 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 20:53:54.811647 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":264,"threads_started":1}
I20260929 20:53:54.814033 8966 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36703
I20260929 20:53:54.814412 8966 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:54.815102 8966 heartbeater.cc:511] Master 127.0.97.62:36703 requested a full tablet report, sending...
I20260929 20:53:54.816381 8643 ts_manager.cc:194] Registered new tserver with Master: 066280fcd5c4400b9e2cc3d51d38e5c6 (127.0.97.2:38105)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:53:54.817130 8643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:53:54.840870 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36703
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:55.147719 8971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:55.147984 8971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:55.148077 8971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:55.155009 8971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:55.155109 8971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:55.155224 8971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:53:55.164350 8971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:55.165488 8971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:55.166705 8971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:55.178889 8979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:55.178893 8977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:55.179178 8976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:55.179629 8971 server_base.cc:1034] running on GCE node
I20260929 20:53:55.180194 8971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:55.181026 8971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:55.182236 8971 hybrid_clock.cc:648] HybridClock initialized: now 1790715235182191 us; error 33 us; skew 500 ppm
I20260929 20:53:55.182539 8971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:55 dist-test-slave-m9g1 krb5kdc[8598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:55.185371 8971 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:53:55.186699 8971 webserver.cc:460] Webserver started at http://127.0.97.3:40931/ using document root <none> and password file <none>
I20260929 20:53:55.187286 8971 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:55.187384 8971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:55.187625 8971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:53:55.190361 8971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "8340705d86d940ad8628bcd7c386821c"
format_stamp: "Formatted at 2026-09-29 20:53:55 on dist-test-slave-m9g1"
server_key: "99a178589d7e843ad7f29dd9be37881d"
server_key_iv: "8ccbc371716ffd9da4c70d3c1955997c"
server_key_version: "encryptionkey@0"
I20260929 20:53:55.191025 8971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8340705d86d940ad8628bcd7c386821c"
format_stamp: "Formatted at 2026-09-29 20:53:55 on dist-test-slave-m9g1"
server_key: "99a178589d7e843ad7f29dd9be37881d"
server_key_iv: "8ccbc371716ffd9da4c70d3c1955997c"
server_key_version: "encryptionkey@0"
I20260929 20:53:55.195497 8971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:53:55.198604 8986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:55.199790 8971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 20:53:55.199930 8971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "8340705d86d940ad8628bcd7c386821c"
format_stamp: "Formatted at 2026-09-29 20:53:55 on dist-test-slave-m9g1"
server_key: "99a178589d7e843ad7f29dd9be37881d"
server_key_iv: "8ccbc371716ffd9da4c70d3c1955997c"
server_key_version: "encryptionkey@0"
I20260929 20:53:55.200089 8971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:55.219316 8971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:55.223179 8971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:55.223430 8971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:55.224279 8971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:55.225479 8971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:55.225549 8971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:55.225675 8971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:55.225728 8971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:55.253496 8971 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43999
I20260929 20:53:55.253537 9098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43999 every 8 connection(s)
I20260929 20:53:55.254716 8971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:53:55.259553 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 8971
I20260929 20:53:55.259711 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:53:55.260206 388 external_mini_cluster.cc:1428] Setting key b38b5272b754ae10fdd8b7f3941da237
Sep 29 20:53:55 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:53:55.273345 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:55.257040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51505 (local address 127.0.97.62:36703)
0929 20:53:55.257211 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:55.257216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:55.258428 (+ 1212us) server_negotiation.cc:407] Connection header received
0929 20:53:55.260761 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:55.260767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:55.260851 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:55.260952 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:55.262763 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:55.263519 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:55.264941 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:55.265337 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:55.265420 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:55.268592 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:55.268599 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:55.268604 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:55.268644 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:55.271003 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:55.271615 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:55.271618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:55.271621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:55.271722 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:55.272038 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:55.272040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:55.272042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:55.272280 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:55.272379 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:55.272964 (+ 585us) server_negotiation.cc:299] Negotiation successful
0929 20:53:55.273121 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:53:55.274573 9101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:55.257536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.3:51505)
0929 20:53:55.258271 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:55.258315 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:55.260460 (+ 2145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:55.261208 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:55.261217 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:55.261951 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:55.262600 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:55.262618 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:55.263641 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:55.263645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:55.264841 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:55.264852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:55.265492 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:55.265496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:55.265676 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:55.266321 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:55.266346 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:55.268342 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:53:55.271114 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:55.271128 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:55.271139 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:55.271525 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:55.271793 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:53:55.271796 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:53:55.271798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:53:55.271958 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:53:55.272363 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:53:55.272368 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:53:55.272667 (+ 299us) client_negotiation.cc:769] Sending connection context
0929 20:53:55.272882 (+ 215us) client_negotiation.cc:240] Negotiation successful
0929 20:53:55.273303 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":272,"threads_started":1}
I20260929 20:53:55.275836 9099 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36703
I20260929 20:53:55.276113 9099 heartbeater.cc:464] Registering TS with master...
I20260929 20:53:55.276773 9099 heartbeater.cc:511] Master 127.0.97.62:36703 requested a full tablet report, sending...
I20260929 20:53:55.278074 8643 ts_manager.cc:194] Registered new tserver with Master: 8340705d86d940ad8628bcd7c386821c (127.0.97.3:43999)
I20260929 20:53:55.278899 8643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51505
I20260929 20:53:55.286913 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260929 20:53:55.362684 8832 heartbeater.cc:503] Master 127.0.97.62:36703 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 20:53:55.405593 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 8704
Sep 29 20:53:55 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:53:55.420504 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:55.406679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45854 (local address 127.0.97.1:35323)
0929 20:53:55.407248 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:55.407255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:55.407285 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:53:55.407343 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:55.407349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:55.407552 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:55.407689 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:55.408879 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:55.409968 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:55.411203 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:55.411746 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:55.411847 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:55.415498 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:55.415509 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:55.415524 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:55.415556 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:55.418142 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:55.418794 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:55.418801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:55.418804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:55.418923 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:55.419207 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:55.419211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:55.419213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:55.419534 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:55.419755 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:55.419955 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 20:53:55.420156 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":274,"threads_started":1}
I20260929 20:53:55.421736 8827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:55.501159 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 8704
I20260929 20:53:55.514671 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 8838
Sep 29 20:53:55 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:53:55.528656 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:55.515487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48662 (local address 127.0.97.2:38105)
0929 20:53:55.515986 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:55.515993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:55.516018 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:53:55.516074 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:55.516080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:55.516296 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:55.516428 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:55.517517 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:55.518543 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:55.519721 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:55.520221 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:55.520309 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:55.524065 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:55.524075 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:55.524089 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:55.524119 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:55.526537 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:55.527105 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:55.527111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:55.527114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:55.527222 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:55.527467 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:55.527470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:55.527472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:55.527772 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:55.528018 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:55.528186 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 20:53:55.528384 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":256,"threads_started":1}
I20260929 20:53:55.529903 8961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:55.608270 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 8838
I20260929 20:53:55.620714 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 8971
Sep 29 20:53:55 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:53:55.635835 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:55.621625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56056 (local address 127.0.97.3:43999)
0929 20:53:55.622214 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:55.622223 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:55.622266 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:53:55.622329 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:55.622336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:55.622538 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:55.622684 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:55.623926 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:55.624934 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:55.626124 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:55.626798 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:55.626914 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:55.630851 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:55.630861 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:55.630876 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:55.630910 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:55.633272 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:55.633887 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:55.633894 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:55.633896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:55.634022 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:55.634381 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:55.634386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:55.634388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:55.634703 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:55.634986 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:55.635289 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 20:53:55.635531 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":282,"threads_started":1}
I20260929 20:53:55.637030 9094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:55.718118 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 8971
I20260929 20:53:55.730715 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 8611
I20260929 20:53:55.731796 8673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:53:55.830302 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 8611
I20260929 20:53:55.846627 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36703
--webserver_interface=127.0.97.62
--webserver_port=38817
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:53:56.150457 9116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:56.150756 9116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:56.150887 9116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:56.157872 9116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:53:56.157989 9116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:56.158070 9116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:56.158143 9116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:53:56.158221 9116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:53:56.167593 9116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:56.168789 9116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:56.170018 9116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:56.180764 9121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:56.180727 9122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:56.180739 9124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:56.181505 9116 server_base.cc:1034] running on GCE node
I20260929 20:53:56.182237 9116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:56.183509 9116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:56.184690 9116 hybrid_clock.cc:648] HybridClock initialized: now 1790715236184638 us; error 42 us; skew 500 ppm
I20260929 20:53:56.184976 9116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:56 dist-test-slave-m9g1 krb5kdc[8598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:56.187707 9116 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:53:56.188993 9116 webserver.cc:460] Webserver started at http://127.0.97.62:38817/ using document root <none> and password file <none>
I20260929 20:53:56.189601 9116 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:56.189723 9116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:56.194630 9116 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:53:56.197077 9131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:56.198314 9116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 20:53:56.198460 9116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "9cbd9e7796184921b3918c65be371e1c"
format_stamp: "Formatted at 2026-09-29 20:53:53 on dist-test-slave-m9g1"
server_key: "06839fccb30a3f4a32078f8149a47c30"
server_key_iv: "36e8e6712de4ad53050475c181a7629e"
server_key_version: "encryptionkey@0"
I20260929 20:53:56.199324 9116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:56.223006 9116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:56.226683 9116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:56.226912 9116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:56.243336 9116 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36703
I20260929 20:53:56.243378 9182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36703 every 8 connection(s)
I20260929 20:53:56.244750 9116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:53:56.251281 9183 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:53:56.253479 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 9116
I20260929 20:53:56.254426 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35323
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:36703
--webserver_port=41929
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:53:56.254974 9183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c: Bootstrap starting.
I20260929 20:53:56.274349 9183 log.cc:824] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:53:56.282859 9183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:53:56.283322 9183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c: Bootstrap complete.
I20260929 20:53:56.292392 9183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbd9e7796184921b3918c65be371e1c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36703 } }
I20260929 20:53:56.293164 9183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd9e7796184921b3918c65be371e1c, State: Initialized, Role: FOLLOWER
I20260929 20:53:56.293761 9183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbd9e7796184921b3918c65be371e1c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36703 } }
I20260929 20:53:56.293958 9183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:53:56.294059 9183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:53:56.294252 9183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:53:56.297660 9183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbd9e7796184921b3918c65be371e1c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36703 } }
I20260929 20:53:56.298095 9183 leader_election.cc:302] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cbd9e7796184921b3918c65be371e1c; no voters:
I20260929 20:53:56.298666 9183 leader_election.cc:288] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:53:56.298789 9187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:53:56.299849 9187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [term 2 LEADER]: Becoming Leader. State: Replica: 9cbd9e7796184921b3918c65be371e1c, State: Running, Role: LEADER
I20260929 20:53:56.300398 9187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbd9e7796184921b3918c65be371e1c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36703 } }
I20260929 20:53:56.300638 9183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:53:56.302873 9189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cbd9e7796184921b3918c65be371e1c. Latest consensus state: current_term: 2 leader_uuid: "9cbd9e7796184921b3918c65be371e1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbd9e7796184921b3918c65be371e1c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36703 } } }
I20260929 20:53:56.303212 9189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:56.303727 9188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9cbd9e7796184921b3918c65be371e1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbd9e7796184921b3918c65be371e1c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36703 } } }
I20260929 20:53:56.304025 9188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c [sys.catalog]: This master's current role is: LEADER
I20260929 20:53:56.305688 9196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:53:56.311077 9196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:53:56.312495 9196 catalog_manager.cc:1259] Loaded cluster ID: aaa97be01e49486daf4ff9ba05eccd91
I20260929 20:53:56.312647 9196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:53:56.319839 9196 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:53:56.320876 9196 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9cbd9e7796184921b3918c65be371e1c: Loaded TSK: 0
I20260929 20:53:56.321910 9196 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 20:53:56.592305 9185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:56.592593 9185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:56.592723 9185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:56.599680 9185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:56.599789 9185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:56.599927 9185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:53:56.611238 9185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41929
--tserver_master_addrs=127.0.97.62:36703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:56.612707 9185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:56.614105 9185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:56.627609 9209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:56.627676 9212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:56.627631 9210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:56.628819 9185 server_base.cc:1034] running on GCE node
I20260929 20:53:56.629376 9185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:56.630282 9185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:56.631487 9185 hybrid_clock.cc:648] HybridClock initialized: now 1790715236631434 us; error 48 us; skew 500 ppm
I20260929 20:53:56.631767 9185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:56 dist-test-slave-m9g1 krb5kdc[8598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715236, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:56.634707 9185 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 20:53:56.636118 9185 webserver.cc:460] Webserver started at http://127.0.97.1:41929/ using document root <none> and password file <none>
I20260929 20:53:56.636747 9185 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:56.636858 9185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:56.642329 9185 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:53:56.644984 9219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:56.646324 9185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:53:56.646462 9185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "04b53758e3e043128c70bfec0ed2e746"
format_stamp: "Formatted at 2026-09-29 20:53:54 on dist-test-slave-m9g1"
server_key: "f2a60bdef02e326df74faf0487f35e5d"
server_key_iv: "1a038c0a9366e6002589ca3144dae0f3"
server_key_version: "encryptionkey@0"
I20260929 20:53:56.647336 9185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:56.677486 9185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:56.681403 9185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:56.681648 9185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:56.682593 9185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:56.683888 9185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:56.683975 9185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:56.684087 9185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:56.684163 9185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:56.716095 9185 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35323
I20260929 20:53:56.716265 9331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35323 every 8 connection(s)
I20260929 20:53:56.717963 9185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:53:56.726223 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 9185
I20260929 20:53:56.726948 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38105
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:36703
--webserver_port=42859
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:53:56 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:53:56 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:53:56.738179 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:56.721914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.1:44207)
0929 20:53:56.722653 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:56.722698 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:56.724266 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:56.725840 (+ 1574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:56.725851 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:56.726603 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:56.727344 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:56.727367 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:56.729503 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:56.729509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:56.731071 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:56.731084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:56.732329 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:56.732336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:56.732548 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:56.733154 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:56.733175 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:56.736862 (+ 3687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":314,"threads_started":1}
W20260929 20:53:56.738261 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:56.721637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44207 (local address 127.0.97.62:36703)
0929 20:53:56.722360 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:56.722376 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:56.722815 (+ 439us) server_negotiation.cc:407] Connection header received
0929 20:53:56.724747 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:56.724767 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:56.725247 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:56.725665 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:56.727563 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:56.729328 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:56.731207 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:56.732133 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:56.732306 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:56.737256 (+ 4950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44207: BlockingRecv error: recv got EOF from 127.0.97.1:44207 (error 108)
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":219,"threads_started":1}
W20260929 20:53:56.738934 9332 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36703 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:36703: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 20:53:56 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:53:56 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:53:56.748056 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:56.739503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.1:48005)
0929 20:53:56.739682 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:56.739701 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:56.739801 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:56.740075 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:56.740079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:56.740444 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:56.740630 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:56.740641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:56.741587 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:56.741592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:56.743091 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:56.743108 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:56.744078 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:56.744085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:56.744270 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:56.744849 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:56.744873 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:56.747855 (+ 2982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 20:53:56.748327 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:56.739607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48005 (local address 127.0.97.62:36703)
0929 20:53:56.739748 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:56.739752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:56.739767 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:53:56.739903 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:56.739906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:56.739960 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:56.740043 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:56.740747 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:56.741463 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:56.743270 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:56.743931 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:56.744032 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:56.748104 (+ 4072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48005: BlockingRecv error: recv got EOF from 127.0.97.1:48005 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 20:53:56 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:53:56 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:53:56.756932 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:56.749590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.1:46983)
0929 20:53:56.749827 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:56.749844 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:56.749933 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:56.750282 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:56.750286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:56.750539 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:56.750669 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:56.750676 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:56.752001 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:56.752004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:56.752963 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:56.752974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:56.753813 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:56.753816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:56.753927 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:56.754487 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:56.754503 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:56.756742 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126}
W20260929 20:53:56.757195 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:56.749835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46983 (local address 127.0.97.62:36703)
0929 20:53:56.749999 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:56.750005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:56.750023 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:53:56.750071 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:56.750076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:56.750156 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:56.750268 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:56.750786 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:56.751879 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:56.753074 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:56.753705 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:56.753818 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:56.756951 (+ 3133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46983: BlockingRecv error: recv got EOF from 127.0.97.1:46983 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 20:53:56.757310 9332 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:53:57.039999 9336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:57.040300 9336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:57.040400 9336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:57.047492 9336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:57.047598 9336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:57.047708 9336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:53:57.057242 9336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42859
--tserver_master_addrs=127.0.97.62:36703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:57.058563 9336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:57.059834 9336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:57.072073 9342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:57.072046 9341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:57.072082 9344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:57.072863 9336 server_base.cc:1034] running on GCE node
I20260929 20:53:57.073398 9336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:57.074282 9336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:57.075472 9336 hybrid_clock.cc:648] HybridClock initialized: now 1790715237075426 us; error 37 us; skew 500 ppm
I20260929 20:53:57.075765 9336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:57 dist-test-slave-m9g1 krb5kdc[8598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715237, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:57.078594 9336 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 20:53:57.079968 9336 webserver.cc:460] Webserver started at http://127.0.97.2:42859/ using document root <none> and password file <none>
I20260929 20:53:57.080610 9336 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:57.080729 9336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:57.085687 9336 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:53:57.088333 9351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:57.089562 9336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:53:57.089706 9336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "066280fcd5c4400b9e2cc3d51d38e5c6"
format_stamp: "Formatted at 2026-09-29 20:53:54 on dist-test-slave-m9g1"
server_key: "d4fd1af990f0a84816ff652587a9d645"
server_key_iv: "24ef0d03e30f89bf1792e32ac99fd248"
server_key_version: "encryptionkey@0"
I20260929 20:53:57.090658 9336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:57.120693 9336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:57.124539 9336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:57.124776 9336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:57.125713 9336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:57.126940 9336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:57.127041 9336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:57.127141 9336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:57.127206 9336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:57.155855 9336 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38105
I20260929 20:53:57.155918 9463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38105 every 8 connection(s)
I20260929 20:53:57.157207 9336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:53:57.159183 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 9336
I20260929 20:53:57.159809 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43999
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:36703
--webserver_port=40931
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:53:57 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:53:57 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:53:57.172333 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:57.159633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50289 (local address 127.0.97.62:36703)
0929 20:53:57.159830 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:57.159836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:57.161097 (+ 1261us) server_negotiation.cc:407] Connection header received
0929 20:53:57.163041 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:57.163047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:57.163134 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:57.163226 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:57.164628 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.165374 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.167054 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.167638 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.167751 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:57.172085 (+ 4334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50289: BlockingRecv error: recv got EOF from 127.0.97.2:50289 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:53:57.172825 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:57.160170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.2:50289)
0929 20:53:57.160930 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:57.160974 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:57.162793 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:57.163429 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:57.163438 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:57.163929 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:57.164472 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:57.164489 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.165532 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.165538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:57.166918 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:57.166930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.167808 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.167813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:57.167993 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:57.168603 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:57.168623 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:57.171861 (+ 3238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":306,"threads_started":1}
W20260929 20:53:57.173440 9464 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36703 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:36703: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 20:53:57 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:53:57 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:53:57.180819 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:57.173963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.2:50567)
0929 20:53:57.174090 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:57.174107 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:57.174217 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:57.174497 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:57.174502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:57.174779 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:57.174898 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:57.174905 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.175809 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.175814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:57.176664 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:57.176674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.177362 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.177368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:57.177521 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:57.178048 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:57.178064 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:57.180575 (+ 2511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 20:53:57.181285 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:57.174065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50567 (local address 127.0.97.62:36703)
0929 20:53:57.174236 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:57.174242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:57.174273 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:53:57.174319 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:57.174324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:57.174384 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:57.174455 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:57.175017 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.175685 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.176796 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.177229 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.177340 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:57.180773 (+ 3433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50567: BlockingRecv error: recv got EOF from 127.0.97.2:50567 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 20:53:57 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:53:57 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:53:57.188184 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:57.181430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.2:60801)
0929 20:53:57.181598 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:57.181612 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:57.181694 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:57.181971 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:57.181976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:57.182282 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:57.182414 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:57.182422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.183298 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.183303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:57.184162 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:57.184172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.184782 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.184788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:57.184934 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:57.185453 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:57.185469 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:57.187931 (+ 2462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 20:53:57.188320 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:57.181592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60801 (local address 127.0.97.62:36703)
0929 20:53:57.181741 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:57.181746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:57.181762 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:53:57.181802 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:57.181805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:57.181854 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:57.181926 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:57.182536 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.183180 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.184281 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.184664 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.184760 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:57.188137 (+ 3377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60801: BlockingRecv error: recv got EOF from 127.0.97.2:60801 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 20:53:57.188467 9464 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:53:57.466642 9467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:53:57.466931 9467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:53:57.467068 9467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:53:57.474157 9467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:53:57.474287 9467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:53:57.474411 9467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:53:57.486297 9467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40931
--tserver_master_addrs=127.0.97.62:36703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:53:57.487517 9467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:53:57.488696 9467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:53:57.500669 9472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:57.500712 9475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:53:57.500641 9473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:53:57.501407 9467 server_base.cc:1034] running on GCE node
I20260929 20:53:57.501994 9467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:53:57.502965 9467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:53:57.504148 9467 hybrid_clock.cc:648] HybridClock initialized: now 1790715237504095 us; error 54 us; skew 500 ppm
I20260929 20:53:57.504436 9467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:53:57 dist-test-slave-m9g1 krb5kdc[8598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715237, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:53:57.507542 9467 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 20:53:57.508868 9467 webserver.cc:460] Webserver started at http://127.0.97.3:40931/ using document root <none> and password file <none>
I20260929 20:53:57.509521 9467 fs_manager.cc:359] Metadata directory not provided
I20260929 20:53:57.509644 9467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:53:57.514791 9467 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:53:57.517292 9482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:57.518567 9467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:53:57.518709 9467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "8340705d86d940ad8628bcd7c386821c"
format_stamp: "Formatted at 2026-09-29 20:53:55 on dist-test-slave-m9g1"
server_key: "99a178589d7e843ad7f29dd9be37881d"
server_key_iv: "8ccbc371716ffd9da4c70d3c1955997c"
server_key_version: "encryptionkey@0"
I20260929 20:53:57.519593 9467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:53:57.546535 9467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:53:57.550156 9467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:53:57.550424 9467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:53:57.551359 9467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:53:57.552595 9467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:53:57.552691 9467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:57.552791 9467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:53:57.552858 9467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:53:57.580755 9467 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43999
I20260929 20:53:57.580816 9594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43999 every 8 connection(s)
I20260929 20:53:57.582064 9467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:53:57.589593 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 9467
Sep 29 20:53:57 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:53:57 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:53:57.597114 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:57.584385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54147 (local address 127.0.97.62:36703)
0929 20:53:57.584544 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:57.584549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:57.585647 (+ 1098us) server_negotiation.cc:407] Connection header received
0929 20:53:57.587373 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:57.587378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:57.587457 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:57.587562 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:57.588910 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.589613 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.591637 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.592270 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.592402 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:57.596872 (+ 4470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54147: BlockingRecv error: recv got EOF from 127.0.97.3:54147 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:53:57.597651 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:57.584849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.3:54147)
0929 20:53:57.585512 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:57.585547 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:57.587132 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:57.587764 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:57.587772 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:57.588239 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:57.588756 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:57.588772 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.589749 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.589752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:57.591505 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:57.591518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.592449 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.592454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:57.592635 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:57.593234 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:57.593254 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:57.596705 (+ 3451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":252,"threads_started":1}
W20260929 20:53:57.598383 9595 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36703 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:36703: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 20:53:57 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:53:57.605170 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:57.590502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41050 (local address 127.0.97.62:36703)
0929 20:53:57.591193 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:57.591198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:57.591216 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:53:57.591261 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:57.591265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:57.591328 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:57.591432 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:57.593089 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.593854 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.595490 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.595904 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.596007 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:57.598696 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:53:57.598706 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:53:57.598725 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:53:57.598785 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:53:57.601995 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:57.602743 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:57.602754 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:57.602760 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:57.602937 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:53:57.603319 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:53:57.603324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:53:57.603327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:53:57.603903 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:53:57.604174 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 20:53:57.604535 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 20:53:57.604876 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":495,"threads_started":1}
Sep 29 20:53:57 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:53:57.606909 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:57.598934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.3:37889)
0929 20:53:57.599083 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:57.599100 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:57.599187 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:57.599462 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:57.599466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:57.599754 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:57.599880 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:57.599887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.600884 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.600889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:57.602280 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:57.602291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.603070 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.603076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:57.603193 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:57.603687 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:57.603707 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:57.606721 (+ 3014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 20:53:57.607160 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:57.598988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37889 (local address 127.0.97.62:36703)
0929 20:53:57.599141 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:57.599146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:57.599161 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:53:57.599288 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:57.599293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:57.599356 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:57.599427 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:57.600002 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.600747 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.602408 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.602943 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.603043 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:57.606947 (+ 3904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37889: BlockingRecv error: recv got EOF from 127.0.97.3:37889 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 20:53:57 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:53:57 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:53:57.614390 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:57.607990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.3:40617)
0929 20:53:57.608150 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:57.608167 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:57.608252 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:57.608539 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:57.608542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:57.608787 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:57.608964 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:57.608972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.609816 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.609819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:57.610734 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:57.610745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.611310 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.611314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:57.611419 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:57.611850 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:57.611864 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:57.614148 (+ 2284us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 20:53:57.614569 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:57.608034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40617 (local address 127.0.97.62:36703)
0929 20:53:57.608238 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:57.608243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:57.608259 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:53:57.608347 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:57.608350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:57.608421 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:57.608504 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:57.609057 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.609707 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.610838 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.611210 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.611302 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:57.614361 (+ 3059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40617: BlockingRecv error: recv got EOF from 127.0.97.3:40617 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260929 20:53:57.614647 9595 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 20:53:57 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:53:57 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:53:57.765843 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:57.757846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.1:39311)
0929 20:53:57.758376 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:57.758397 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:57.758530 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:57.758849 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:57.758863 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:57.759429 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:57.759667 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:57.759683 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.760759 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.760763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:57.761683 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:57.761694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.762350 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.762354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:57.762469 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:57.762969 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:57.762984 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:57.765545 (+ 2561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":248,"threads_started":1}
W20260929 20:53:57.766083 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:57.757916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39311 (local address 127.0.97.62:36703)
0929 20:53:57.758066 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:57.758070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:57.758486 (+ 416us) server_negotiation.cc:407] Connection header received
0929 20:53:57.758644 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:57.758649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:57.758710 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:57.758793 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:57.759806 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.760636 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.761790 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:57.762212 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:57.762316 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:57.765755 (+ 3439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39311: BlockingRecv error: recv got EOF from 127.0.97.1:39311 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 20:53:58 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:53:58 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:53:58.196924 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:58.189015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.2:35941)
0929 20:53:58.189477 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:58.189492 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:58.189604 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:58.189870 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:58.189873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:58.190308 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:58.190494 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:58.190502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:58.191378 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:58.191381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:58.192327 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:58.192338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:58.193162 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:58.193169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:58.193302 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:58.193971 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:58.193994 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:58.196714 (+ 2720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":244,"threads_started":1}
W20260929 20:53:58.197103 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:58.189067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35941 (local address 127.0.97.62:36703)
0929 20:53:58.189252 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:58.189257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:58.189559 (+ 302us) server_negotiation.cc:407] Connection header received
0929 20:53:58.189700 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:58.189704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:58.189761 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:58.189843 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:58.190599 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:58.191265 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:58.192462 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:58.192990 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:58.193105 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:58.196886 (+ 3781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35941: BlockingRecv error: recv got EOF from 127.0.97.2:35941 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 20:53:58 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:53:58 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:53:58.623101 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:58.615293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.3:47541)
0929 20:53:58.615771 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:58.615786 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:58.615923 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:58.616237 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:58.616242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:58.616777 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:58.617004 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:58.617014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:58.617946 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:58.617949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:58.618922 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:58.618934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:58.619586 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:58.619590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:58.619716 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:58.620242 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:58.620258 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:58.622809 (+ 2551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":224,"threads_started":1}
W20260929 20:53:58.623260 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:58.615520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47541 (local address 127.0.97.62:36703)
0929 20:53:58.615719 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:58.615725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:58.615868 (+ 143us) server_negotiation.cc:407] Connection header received
0929 20:53:58.616029 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:58.616033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:58.616100 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:58.616177 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:58.617130 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:58.617839 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:58.619073 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:58.619458 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:58.619547 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:58.623030 (+ 3483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47541: BlockingRecv error: recv got EOF from 127.0.97.3:47541 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 20:53:58 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:53:58 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:53:58.774483 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:58.766700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.1:58623)
0929 20:53:58.767161 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:58.767179 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:58.767298 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:58.767605 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:58.767609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:58.767982 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:58.768122 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:58.768131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:58.769230 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:58.769235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:58.770454 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:58.770468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:58.771066 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:58.771070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:58.771176 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:58.771728 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:58.771745 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:58.774261 (+ 2516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":255,"threads_started":1}
W20260929 20:53:58.774593 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:58.766764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58623 (local address 127.0.97.62:36703)
0929 20:53:58.766972 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:58.766978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:58.767251 (+ 273us) server_negotiation.cc:407] Connection header received
0929 20:53:58.767405 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:58.767411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:58.767482 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:58.767556 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:58.768248 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:58.769092 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:58.770568 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:58.770948 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:58.771075 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:58.774402 (+ 3327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58623: BlockingRecv error: recv got EOF from 127.0.97.1:58623 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 20:53:59 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:53:59 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:53:59.205758 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:59.197734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.2:48065)
0929 20:53:59.198240 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:59.198255 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:59.198387 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:59.198707 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:59.198712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:59.199129 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:59.199268 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:59.199277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:59.200247 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:59.200251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:59.201208 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:59.201221 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:59.201892 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:59.201898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:59.202078 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:59.202672 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:59.202690 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:59.205449 (+ 2759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":279,"threads_started":1}
W20260929 20:53:59.205894 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:59.197824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48065 (local address 127.0.97.62:36703)
0929 20:53:59.198018 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:59.198025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:59.198339 (+ 314us) server_negotiation.cc:407] Connection header received
0929 20:53:59.198498 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:59.198502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:59.198572 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:59.198651 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:59.199401 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:59.200119 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:59.201331 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:59.201751 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:59.201879 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:59.205687 (+ 3808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48065: BlockingRecv error: recv got EOF from 127.0.97.2:48065 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 29 20:53:59 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:53:59 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:53:59.631788 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:59.623998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.3:49367)
0929 20:53:59.624489 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:59.624504 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:59.624627 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:59.624957 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:59.624959 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:59.625354 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:59.625499 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:59.625508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:59.626651 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:59.626655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:59.627532 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:59.627551 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:59.628167 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:59.628171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:59.628275 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:59.628781 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:59.628797 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:59.631557 (+ 2760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":252,"threads_started":1}
W20260929 20:53:59.631948 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:59.624134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49367 (local address 127.0.97.62:36703)
0929 20:53:59.624302 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:59.624306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:59.624580 (+ 274us) server_negotiation.cc:407] Connection header received
0929 20:53:59.624760 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:59.624764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:59.624846 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:59.624940 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:59.625673 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:59.626522 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:59.627659 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:59.628058 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:59.628149 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:59.631716 (+ 3567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49367: BlockingRecv error: recv got EOF from 127.0.97.3:49367 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 20:53:59 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:53:59 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:53:59.782635 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:53:59.775344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.1:58985)
0929 20:53:59.775775 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 20:53:59.775794 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:53:59.775905 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:53:59.776160 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:53:59.776163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:53:59.776481 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:53:59.776646 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:59.776654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:59.777560 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:59.777563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:59.778481 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:53:59.778493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:53:59.779049 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:53:59.779052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:53:59.779161 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:59.779682 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:53:59.779697 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:53:59.782306 (+ 2609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":228,"threads_started":1}
W20260929 20:53:59.782740 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:53:59.775400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58985 (local address 127.0.97.62:36703)
0929 20:53:59.775622 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 20:53:59.775629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:53:59.775863 (+ 234us) server_negotiation.cc:407] Connection header received
0929 20:53:59.775995 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:53:59.775998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:53:59.776056 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:53:59.776131 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:53:59.776747 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:59.777441 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:59.778581 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:53:59.778949 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:53:59.779034 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:53:59.782520 (+ 3486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58985: BlockingRecv error: recv got EOF from 127.0.97.1:58985 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 29 20:54:00 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:00 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:00.214458 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:00.206634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.2:48723)
0929 20:54:00.207107 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:00.207122 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:00.207249 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:00.207569 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:00.207573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:00.207913 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:00.208070 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:00.208078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:00.209078 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:00.209082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:00.209959 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:00.209971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:00.210820 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:00.210826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:00.210998 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:00.211565 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:00.211583 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:00.214168 (+ 2585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":241,"threads_started":1}
W20260929 20:54:00.214579 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:00.206722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48723 (local address 127.0.97.62:36703)
0929 20:54:00.206908 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:00.206914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:00.207203 (+ 289us) server_negotiation.cc:407] Connection header received
0929 20:54:00.207357 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:00.207363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:00.207439 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:00.207518 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:00.208205 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:00.208943 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:00.210094 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:00.210682 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:00.210813 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:00.214390 (+ 3577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48723: BlockingRecv error: recv got EOF from 127.0.97.2:48723 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 20:54:00 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:00 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:00.640383 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:00.632611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.3:53939)
0929 20:54:00.633142 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:00.633159 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:00.633277 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:00.633541 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:00.633544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:00.633847 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:00.634014 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:00.634022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:00.634923 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:00.634927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:00.635788 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:00.635799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:00.636471 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:00.636474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:00.636589 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:00.637243 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:00.637265 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:00.640151 (+ 2886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":289,"threads_started":1}
W20260929 20:54:00.640534 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:00.632713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53939 (local address 127.0.97.62:36703)
0929 20:54:00.632876 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:00.632881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:00.633218 (+ 337us) server_negotiation.cc:407] Connection header received
0929 20:54:00.633360 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:00.633364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:00.633432 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:00.633511 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:00.634121 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:00.634825 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:00.635919 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:00.636348 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:00.636475 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:00.640349 (+ 3874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53939: BlockingRecv error: recv got EOF from 127.0.97.3:53939 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 20:54:00 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:00 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:00.791263 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:00.783555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.1:45637)
0929 20:54:00.784000 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:00.784018 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:00.784125 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:00.784387 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:00.784391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:00.784703 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:00.784843 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:00.784850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:00.785855 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:00.785859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:00.786818 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:00.786831 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:00.787422 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:00.787425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:00.787544 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:00.788137 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:00.788154 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:00.791047 (+ 2893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":217,"threads_started":1}
W20260929 20:54:00.791457 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:00.783675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45637 (local address 127.0.97.62:36703)
0929 20:54:00.783874 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:00.783879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:00.784084 (+ 205us) server_negotiation.cc:407] Connection header received
0929 20:54:00.784217 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:00.784221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:00.784279 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:00.784354 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:00.784948 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:00.785751 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:00.786931 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:00.787321 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:00.787407 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:00.791225 (+ 3818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45637: BlockingRecv error: recv got EOF from 127.0.97.1:45637 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 20:54:01 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:01 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:01.223133 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:01.215333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.2:56517)
0929 20:54:01.215847 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:01.215866 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:01.216002 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:01.216376 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:01.216381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:01.216856 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:01.217047 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:01.217058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:01.218013 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:01.218016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:01.218988 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:01.219000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:01.219600 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:01.219603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:01.219744 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:01.220264 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:01.220279 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:01.222916 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":281,"threads_started":1}
W20260929 20:54:01.223279 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:01.215430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56517 (local address 127.0.97.62:36703)
0929 20:54:01.215634 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:01.215641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:01.215950 (+ 309us) server_negotiation.cc:407] Connection header received
0929 20:54:01.216128 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:01.216133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:01.216236 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:01.216333 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:01.217184 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:01.217910 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:01.219103 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:01.219500 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:01.219586 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:01.223083 (+ 3497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56517: BlockingRecv error: recv got EOF from 127.0.97.2:56517 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 20:54:01 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:01 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:01.648340 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:01.641159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.3:58943)
0929 20:54:01.641576 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:01.641592 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:01.641708 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:01.641962 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:01.641965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:01.642352 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:01.642503 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:01.642523 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:01.643464 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:01.643467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:01.644391 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:01.644400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:01.644957 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:01.644961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:01.645082 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:01.645599 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:01.645615 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:01.648123 (+ 2508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":216,"threads_started":1}
W20260929 20:54:01.648509 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:01.641230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58943 (local address 127.0.97.62:36703)
0929 20:54:01.641424 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:01.641429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:01.641654 (+ 225us) server_negotiation.cc:407] Connection header received
0929 20:54:01.641799 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:01.641802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:01.641859 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:01.641934 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:01.642634 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:01.643370 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:01.644487 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:01.644855 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:01.644940 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:01.648313 (+ 3373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58943: BlockingRecv error: recv got EOF from 127.0.97.3:58943 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 20:54:01 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:01 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:01.799822 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:01.792129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.1:36053)
0929 20:54:01.792590 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:01.792616 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:01.792766 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:01.793086 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:01.793090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:01.793558 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:01.793769 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:01.793781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:01.794984 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:01.794988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:01.795985 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:01.795997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:01.796584 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:01.796587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:01.796694 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:01.797237 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:01.797254 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:01.799596 (+ 2342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":200,"threads_started":1}
W20260929 20:54:01.799955 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:01.792201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36053 (local address 127.0.97.62:36703)
0929 20:54:01.792409 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:01.792415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:01.792694 (+ 279us) server_negotiation.cc:407] Connection header received
0929 20:54:01.792879 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:01.792884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:01.792957 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:01.793037 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:01.793912 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:01.794880 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:01.796090 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:01.796485 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:01.796568 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:01.799776 (+ 3208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36053: BlockingRecv error: recv got EOF from 127.0.97.1:36053 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 20:54:02 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:02 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:02.239004 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:02.224079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.2:51607)
0929 20:54:02.224572 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:02.224589 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:02.224698 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:02.224944 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:02.224947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:02.225289 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:02.225444 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:02.225451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:02.226439 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:02.226443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:02.227358 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:02.227370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:02.227969 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:02.227972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:02.228087 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:02.236117 (+ 8030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:02.236136 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:02.238773 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":230,"threads_started":1}
W20260929 20:54:02.239187 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:02.224256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51607 (local address 127.0.97.62:36703)
0929 20:54:02.224442 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:02.224448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:02.224661 (+ 213us) server_negotiation.cc:407] Connection header received
0929 20:54:02.224780 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:02.224783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:02.224839 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:02.224912 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:02.225577 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:02.226347 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:02.227462 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:02.227868 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:02.227954 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:02.238963 (+ 11009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51607: BlockingRecv error: recv got EOF from 127.0.97.2:51607 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 20:54:02 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:02 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:02.656970 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:02.649174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.3:40971)
0929 20:54:02.649645 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:02.649660 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:02.649765 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:02.650034 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:02.650038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:02.650428 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:02.650592 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:02.650601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:02.651643 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:02.651649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:02.652702 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:02.652714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:02.653398 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:02.653402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:02.653508 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:02.654162 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:02.654179 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:02.656739 (+ 2560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":225,"threads_started":1}
W20260929 20:54:02.657114 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:02.649271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40971 (local address 127.0.97.62:36703)
0929 20:54:02.649417 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:02.649421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:02.649726 (+ 305us) server_negotiation.cc:407] Connection header received
0929 20:54:02.649852 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:02.649856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:02.649913 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:02.649990 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:02.650705 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:02.651516 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:02.652851 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:02.653273 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:02.653379 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:02.656915 (+ 3536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40971: BlockingRecv error: recv got EOF from 127.0.97.3:40971 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 20:54:02 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:02 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:02.808399 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:02.800582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.1:55243)
0929 20:54:02.801072 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:02.801092 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:02.801229 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:02.801539 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:02.801542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:02.801894 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:02.802078 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:02.802092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:02.803440 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:02.803444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:02.804324 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:02.804335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:02.805008 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:02.805012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:02.805124 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:02.805685 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:02.805702 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:02.808194 (+ 2492us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":267,"threads_started":1}
W20260929 20:54:02.808629 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:02.800637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55243 (local address 127.0.97.62:36703)
0929 20:54:02.800777 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:02.800781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:02.801152 (+ 371us) server_negotiation.cc:407] Connection header received
0929 20:54:02.801367 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:02.801371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:02.801429 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:02.801513 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:02.802237 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:02.803315 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:02.804430 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:02.804885 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:02.804973 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:02.808378 (+ 3405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55243: BlockingRecv error: recv got EOF from 127.0.97.1:55243 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 20:54:03 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:03 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:03.247687 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:03.239796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.2:45777)
0929 20:54:03.240271 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:03.240287 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:03.240409 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:03.240726 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:03.240731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:03.241106 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:03.241266 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:03.241274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:03.242253 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:03.242259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:03.243140 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:03.243151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:03.243842 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:03.243848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:03.244009 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:03.244553 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:03.244570 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:03.247403 (+ 2833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":273,"threads_started":1}
W20260929 20:54:03.247890 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:03.239882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45777 (local address 127.0.97.62:36703)
0929 20:54:03.240069 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:03.240075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:03.240364 (+ 289us) server_negotiation.cc:407] Connection header received
0929 20:54:03.240551 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:03.240554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:03.240612 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:03.240687 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:03.241412 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:03.242108 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:03.243276 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:03.243690 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:03.243804 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:03.247651 (+ 3847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45777: BlockingRecv error: recv got EOF from 127.0.97.2:45777 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 20:54:03 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:03 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:03.665699 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:03.657811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.3:45761)
0929 20:54:03.658333 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:03.658349 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:03.658494 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:03.658784 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:03.658789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:03.659155 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:03.659309 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:03.659318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:03.660346 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:03.660350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:03.661236 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:03.661246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:03.661940 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:03.661946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:03.662102 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:03.662697 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:03.662713 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:03.665411 (+ 2698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":271,"threads_started":1}
W20260929 20:54:03.665853 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:03.657858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45761 (local address 127.0.97.62:36703)
0929 20:54:03.658015 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:03.658019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:03.658445 (+ 426us) server_negotiation.cc:407] Connection header received
0929 20:54:03.658598 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:03.658603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:03.658664 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:03.658738 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:03.659450 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:03.660217 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:03.661364 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:03.661786 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:03.661908 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:03.665607 (+ 3699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45761: BlockingRecv error: recv got EOF from 127.0.97.3:45761 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 20:54:03 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:03 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:03.816773 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:03.809214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.1:43743)
0929 20:54:03.809762 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:03.809780 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:03.809892 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:03.810135 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:03.810138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:03.810539 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:03.810694 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:03.810702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:03.811597 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:03.811600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:03.812485 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:03.812495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:03.813092 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:03.813096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:03.813195 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:03.813684 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:03.813699 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:03.816477 (+ 2778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":241,"threads_started":1}
W20260929 20:54:03.816898 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:03.809298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43743 (local address 127.0.97.62:36703)
0929 20:54:03.809459 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:03.809463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:03.809840 (+ 377us) server_negotiation.cc:407] Connection header received
0929 20:54:03.809975 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:03.809979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:03.810034 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:03.810108 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:03.810798 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:03.811489 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:03.812588 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:03.812989 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:03.813073 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:03.816683 (+ 3610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43743: BlockingRecv error: recv got EOF from 127.0.97.1:43743 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 20:54:04 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:04 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:04.256973 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:04.248755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.2:42725)
0929 20:54:04.249653 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:04.249671 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:04.249804 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:04.250206 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:04.250212 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:04.250646 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:04.250804 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:04.250813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:04.251816 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:04.251821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:04.252701 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:04.252712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:04.253395 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:04.253402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:04.253542 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:04.254090 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:04.254107 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:04.256718 (+ 2611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":549,"threads_started":1}
W20260929 20:54:04.257120 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:04.248798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42725 (local address 127.0.97.62:36703)
0929 20:54:04.248999 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:04.249005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:04.249762 (+ 757us) server_negotiation.cc:407] Connection header received
0929 20:54:04.249971 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:04.249977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:04.250053 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:04.250132 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:04.250935 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:04.251691 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:04.252837 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:04.253253 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:04.253356 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:04.256921 (+ 3565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42725: BlockingRecv error: recv got EOF from 127.0.97.2:42725 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 29 20:54:04 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:04 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:04.674566 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:04.666675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.3:43643)
0929 20:54:04.667271 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:04.667286 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:04.667417 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:04.667718 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:04.667737 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:04.668110 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:04.668267 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:04.668277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:04.669250 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:04.669256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:04.670144 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:04.670156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:04.670875 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:04.670879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:04.670986 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:04.671497 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:04.671513 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:04.674243 (+ 2730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":242,"threads_started":1}
W20260929 20:54:04.674621 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:04.666775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43643 (local address 127.0.97.62:36703)
0929 20:54:04.666970 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:04.666976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:04.667374 (+ 398us) server_negotiation.cc:407] Connection header received
0929 20:54:04.667529 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:04.667534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:04.667594 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:04.667672 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:04.668406 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:04.669116 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:04.670310 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:04.670749 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:04.670849 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:04.674429 (+ 3580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43643: BlockingRecv error: recv got EOF from 127.0.97.3:43643 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 20:54:04 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:04 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:04.825278 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:04.817727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.1:47715)
0929 20:54:04.818174 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:04.818238 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:04.818358 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:04.818615 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:04.818617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:04.818989 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:04.819128 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:04.819136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:04.820078 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:04.820082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:04.820979 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:04.820989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:04.821606 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:04.821610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:04.821721 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:04.822332 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:04.822350 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:04.824999 (+ 2649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":248,"threads_started":1}
W20260929 20:54:04.825491 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:04.817882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47715 (local address 127.0.97.62:36703)
0929 20:54:04.818057 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:04.818062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:04.818310 (+ 248us) server_negotiation.cc:407] Connection header received
0929 20:54:04.818457 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:04.818460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:04.818517 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:04.818588 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:04.819250 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:04.819950 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:04.821081 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:04.821477 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:04.821562 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:04.825197 (+ 3635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47715: BlockingRecv error: recv got EOF from 127.0.97.1:47715 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 20:54:05 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:05 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:05.265199 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:05.257968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.2:58757)
0929 20:54:05.258465 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:05.258482 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:05.258594 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:05.258848 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:05.258851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:05.259216 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:05.259359 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:05.259366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:05.260431 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:05.260434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:05.261307 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:05.261316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:05.261941 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:05.261945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:05.262058 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:05.262578 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:05.262593 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:05.264968 (+ 2375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":290,"threads_started":1}
W20260929 20:54:05.265302 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:05.258058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58757 (local address 127.0.97.62:36703)
0929 20:54:05.258230 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:05.258237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:05.258546 (+ 309us) server_negotiation.cc:407] Connection header received
0929 20:54:05.258683 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:05.258686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:05.258743 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:05.258820 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:05.259464 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:05.260323 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:05.261412 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:05.261815 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:05.261899 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:05.265137 (+ 3238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58757: BlockingRecv error: recv got EOF from 127.0.97.2:58757 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 20:54:05 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:05 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:05.682893 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:05.675539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.3:39411)
0929 20:54:05.675994 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:05.676009 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:05.676123 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:05.676443 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:05.676447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:05.676869 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:05.677019 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:05.677028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:05.677996 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:05.678000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:05.678920 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:05.678930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:05.679608 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:05.679612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:05.679724 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:05.680244 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:05.680259 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:05.682684 (+ 2425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":247,"threads_started":1}
W20260929 20:54:05.683097 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:05.675749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39411 (local address 127.0.97.62:36703)
0929 20:54:05.675932 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:05.675937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:05.676092 (+ 155us) server_negotiation.cc:407] Connection header received
0929 20:54:05.676228 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:05.676234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:05.676303 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:05.676393 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:05.677139 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:05.677876 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:05.679062 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:05.679465 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:05.679565 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:05.682858 (+ 3293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39411: BlockingRecv error: recv got EOF from 127.0.97.3:39411 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 20:54:05 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:05 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:05.833698 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:05.826087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.1:45875)
0929 20:54:05.826552 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:05.826575 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:05.826690 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:05.826959 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:05.826962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:05.827426 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:05.827644 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:05.827655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:05.828793 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:05.828797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:05.829718 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:05.829730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:05.830356 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:05.830360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:05.830470 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:05.830970 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:05.830987 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:05.833464 (+ 2477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":254,"threads_started":1}
W20260929 20:54:05.833834 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:05.826156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45875 (local address 127.0.97.62:36703)
0929 20:54:05.826408 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:05.826414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:05.826678 (+ 264us) server_negotiation.cc:407] Connection header received
0929 20:54:05.826787 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:05.826791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:05.826851 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:05.826933 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:05.827751 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:05.828684 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:05.829823 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:05.830252 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:05.830348 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:05.833647 (+ 3299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45875: BlockingRecv error: recv got EOF from 127.0.97.1:45875 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 20:54:06 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:06 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:06.273762 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:06.266077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.2:35841)
0929 20:54:06.266543 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:06.266560 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:06.266668 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:06.266907 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:06.266910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:06.267257 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:06.267408 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:06.267415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:06.268347 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:06.268351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:06.269287 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:06.269297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:06.270009 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:06.270014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:06.270137 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:06.270684 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:06.270700 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:06.273443 (+ 2743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":241,"threads_started":1}
W20260929 20:54:06.273849 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:06.266129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35841 (local address 127.0.97.62:36703)
0929 20:54:06.266301 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:06.266305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:06.266624 (+ 319us) server_negotiation.cc:407] Connection header received
0929 20:54:06.266758 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:06.266761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:06.266818 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:06.266893 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:06.267515 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:06.268251 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:06.269434 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:06.269845 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:06.269980 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:06.273628 (+ 3648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35841: BlockingRecv error: recv got EOF from 127.0.97.2:35841 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 20:54:06 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:06 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:06.691804 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:06.683895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.3:45257)
0929 20:54:06.684552 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:06.684571 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:06.684699 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:06.685034 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:06.685040 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:06.685406 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:06.685590 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:06.685604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:06.686645 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:06.686651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:06.687563 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:06.687575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:06.688234 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:06.688240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:06.688396 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:06.688902 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:06.688919 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:06.691576 (+ 2657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":385,"threads_started":1}
W20260929 20:54:06.691941 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:06.683960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45257 (local address 127.0.97.62:36703)
0929 20:54:06.684163 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:06.684169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:06.684666 (+ 497us) server_negotiation.cc:407] Connection header received
0929 20:54:06.684801 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:06.684805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:06.684884 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:06.684973 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:06.685728 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:06.686514 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:06.687694 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:06.688093 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:06.688195 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:06.691756 (+ 3561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45257: BlockingRecv error: recv got EOF from 127.0.97.3:45257 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 20:54:06 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:06 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:06.842824 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:06.834609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.1:50217)
0929 20:54:06.835124 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:06.835144 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:06.835275 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:06.835586 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:06.835590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:06.835958 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:06.836137 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:06.836149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:06.837352 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:06.837358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:06.838332 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:06.838344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:06.839082 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:06.839087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:06.839210 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:06.839770 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:06.839787 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:06.842495 (+ 2708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":293,"threads_started":1}
W20260929 20:54:06.842907 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:06.834698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50217 (local address 127.0.97.62:36703)
0929 20:54:06.834971 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:06.834977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:06.835230 (+ 253us) server_negotiation.cc:407] Connection header received
0929 20:54:06.835389 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:06.835395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:06.835459 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:06.835538 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:06.836290 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:06.837217 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:06.838446 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:06.838935 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:06.839057 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:06.842688 (+ 3631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50217: BlockingRecv error: recv got EOF from 127.0.97.1:50217 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 29 20:54:07 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:07 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:07.287565 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:07.274678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.2:59945)
0929 20:54:07.275179 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:07.275194 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:07.275317 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:07.275628 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:07.275633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:07.276050 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:07.276188 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:07.276196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:07.277157 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:07.277163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:07.278071 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:07.278082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:07.278786 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:07.278792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:07.278958 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:07.284617 (+ 5659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:07.284642 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:07.287350 (+ 2708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":285,"threads_started":1}
W20260929 20:54:07.287752 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:07.274817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59945 (local address 127.0.97.62:36703)
0929 20:54:07.275045 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:07.275051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:07.275273 (+ 222us) server_negotiation.cc:407] Connection header received
0929 20:54:07.275445 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:07.275449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:07.275506 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:07.275583 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:07.276328 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:07.277037 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:07.278222 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:07.278650 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:07.278754 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:07.287529 (+ 8775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59945: BlockingRecv error: recv got EOF from 127.0.97.2:59945 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 20:54:07 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:07 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:07.700673 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:07.692742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.3:54033)
0929 20:54:07.693228 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:07.693243 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:07.693353 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:07.693608 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:07.693611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:07.694055 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:07.694222 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:07.694231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:07.695318 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:07.695322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:07.696227 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:07.696238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:07.696940 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:07.696945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:07.697074 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:07.697686 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:07.697705 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:07.700319 (+ 2614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":262,"threads_started":1}
W20260929 20:54:07.700780 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:07.692892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54033 (local address 127.0.97.62:36703)
0929 20:54:07.693119 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:07.693124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:07.693309 (+ 185us) server_negotiation.cc:407] Connection header received
0929 20:54:07.693439 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:07.693443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:07.693500 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:07.693583 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:07.694374 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:07.695191 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:07.696356 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:07.696787 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:07.696917 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:07.700553 (+ 3636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54033: BlockingRecv error: recv got EOF from 127.0.97.3:54033 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Sep 29 20:54:07 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:07 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:07.851153 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:07.843642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.1:35893)
0929 20:54:07.844156 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:07.844175 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:07.844282 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:07.844569 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:07.844572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:07.844937 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:07.845104 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:07.845112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:07.846011 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:07.846014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:07.846934 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:07.846945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:07.847579 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:07.847583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:07.847720 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:07.848302 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:07.848320 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:07.850922 (+ 2602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":297,"threads_started":1}
W20260929 20:54:07.851291 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:07.843740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35893 (local address 127.0.97.62:36703)
0929 20:54:07.843915 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:07.843919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:07.844241 (+ 322us) server_negotiation.cc:407] Connection header received
0929 20:54:07.844393 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:07.844397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:07.844458 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:07.844540 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:07.845214 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:07.845913 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:07.847051 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:07.847467 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:07.847558 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:07.851090 (+ 3532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35893: BlockingRecv error: recv got EOF from 127.0.97.1:35893 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 20:54:08 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:08 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:08.296193 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:08.288342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.2:58713)
0929 20:54:08.288799 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:08.288814 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:08.288939 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:08.289245 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:08.289250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:08.289661 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:08.289836 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:08.289845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:08.290890 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:08.290896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:08.291779 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:08.291790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:08.292447 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:08.292453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:08.292612 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:08.293140 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:08.293157 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:08.295883 (+ 2726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":232,"threads_started":1}
W20260929 20:54:08.296314 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:08.288461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58713 (local address 127.0.97.62:36703)
0929 20:54:08.288651 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:08.288658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:08.288889 (+ 231us) server_negotiation.cc:407] Connection header received
0929 20:54:08.289054 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:08.289059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:08.289125 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:08.289203 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:08.289968 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:08.290760 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:08.291903 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:08.292312 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:08.292413 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:08.296087 (+ 3674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58713: BlockingRecv error: recv got EOF from 127.0.97.2:58713 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 20:54:08 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:08 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:08.716564 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:08.701529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.3:44629)
0929 20:54:08.702027 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:08.702043 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:08.702154 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:08.702493 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:08.702497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:08.702897 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:08.703068 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:08.703077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:08.704054 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:08.704058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:08.704986 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:08.704999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:08.705604 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:08.705610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:08.705723 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:08.713502 (+ 7779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:08.713522 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:08.716257 (+ 2735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":275,"threads_started":1}
W20260929 20:54:08.716706 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:08.701637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44629 (local address 127.0.97.62:36703)
0929 20:54:08.701784 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:08.701788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:08.702106 (+ 318us) server_negotiation.cc:407] Connection header received
0929 20:54:08.702281 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:08.702301 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:08.702361 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:08.702445 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:08.703171 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:08.703938 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:08.705085 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:08.705489 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:08.705579 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:08.716469 (+ 10890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44629: BlockingRecv error: recv got EOF from 127.0.97.3:44629 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 29 20:54:08 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:08 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:08.859776 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:08.851981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.1:57643)
0929 20:54:08.852468 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:08.852491 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:08.852600 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:08.852866 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:08.852869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:08.853224 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:08.853369 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:08.853377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:08.854289 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:08.854292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:08.855583 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:08.855594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:08.856266 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:08.856270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:08.856389 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:08.856908 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:08.856923 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:08.859471 (+ 2548us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":262,"threads_started":1}
W20260929 20:54:08.859894 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:08.852017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57643 (local address 127.0.97.62:36703)
0929 20:54:08.852187 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:08.852191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:08.852552 (+ 361us) server_negotiation.cc:407] Connection header received
0929 20:54:08.852693 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:08.852697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:08.852754 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:08.852829 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:08.853474 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:08.854157 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:08.855702 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:08.856160 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:08.856250 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:08.859669 (+ 3419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57643: BlockingRecv error: recv got EOF from 127.0.97.1:57643 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 20:54:09 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:09 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:09.304904 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:09.297053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.2:34067)
0929 20:54:09.297596 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:09.297614 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:09.297762 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:09.298083 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:09.298086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:09.298541 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:09.298701 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:09.298711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:09.299594 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:09.299597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:09.300521 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:09.300533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:09.301208 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:09.301212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:09.301330 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:09.301978 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:09.302001 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:09.304585 (+ 2584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":323,"threads_started":1}
W20260929 20:54:09.305013 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:09.297126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34067 (local address 127.0.97.62:36703)
0929 20:54:09.297332 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:09.297337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:09.297724 (+ 387us) server_negotiation.cc:407] Connection header received
0929 20:54:09.297874 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:09.297880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:09.297952 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:09.298039 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:09.298814 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:09.299492 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:09.300627 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:09.301099 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:09.301188 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:09.304784 (+ 3596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34067: BlockingRecv error: recv got EOF from 127.0.97.2:34067 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 20:54:09 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:09 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:09.725436 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:09.717408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.3:36009)
0929 20:54:09.717894 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:09.717911 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:09.718020 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:09.718320 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:09.718323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:09.718675 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:09.718822 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:09.718830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:09.719815 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:09.719820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:09.720745 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:09.720757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:09.721458 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:09.721462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:09.721616 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:09.722210 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:09.722235 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:09.725164 (+ 2929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":267,"threads_started":1}
W20260929 20:54:09.725622 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:09.717484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36009 (local address 127.0.97.62:36703)
0929 20:54:09.717675 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:09.717681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:09.717980 (+ 299us) server_negotiation.cc:407] Connection header received
0929 20:54:09.718111 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:09.718117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:09.718209 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:09.718305 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:09.718952 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:09.719669 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:09.720889 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:09.721326 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:09.721433 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:09.725367 (+ 3934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36009: BlockingRecv error: recv got EOF from 127.0.97.3:36009 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 20:54:09 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:09 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:09.868443 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:09.860683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.1:48699)
0929 20:54:09.861175 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:09.861194 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:09.861346 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:09.861657 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:09.861661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:09.862043 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:09.862236 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:09.862248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:09.863219 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:09.863223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:09.864137 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:09.864148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:09.864754 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:09.864757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:09.864864 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:09.865424 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:09.865445 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:09.868224 (+ 2779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":239,"threads_started":1}
W20260929 20:54:09.868592 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:09.860797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48699 (local address 127.0.97.62:36703)
0929 20:54:09.860984 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:09.860990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:09.861267 (+ 277us) server_negotiation.cc:407] Connection header received
0929 20:54:09.861450 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:09.861455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:09.861529 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:09.861616 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:09.862373 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:09.863110 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:09.864238 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:09.864627 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:09.864714 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:09.868381 (+ 3667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48699: BlockingRecv error: recv got EOF from 127.0.97.1:48699 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 20:54:10 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:10 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:10.313025 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:10.305803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.2:53589)
0929 20:54:10.306350 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:10.306370 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:10.306488 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:10.306728 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:10.306731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:10.307086 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:10.307236 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:10.307243 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:10.308156 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:10.308159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:10.309040 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:10.309050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:10.309649 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:10.309655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:10.309760 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:10.310305 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:10.310320 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:10.312805 (+ 2485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":260,"threads_started":1}
W20260929 20:54:10.313259 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:10.305911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53589 (local address 127.0.97.62:36703)
0929 20:54:10.306064 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:10.306068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:10.306431 (+ 363us) server_negotiation.cc:407] Connection header received
0929 20:54:10.306580 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:10.306584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:10.306640 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:10.306713 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:10.307334 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:10.308058 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:10.309139 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:10.309509 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:10.309609 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:10.313043 (+ 3434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53589: BlockingRecv error: recv got EOF from 127.0.97.2:53589 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 20:54:10 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:10 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:10.734455 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:10.726522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.3:49643)
0929 20:54:10.727027 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:10.727044 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:10.727159 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:10.727463 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:10.727467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:10.727818 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:10.727976 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:10.727999 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:10.728958 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:10.728961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:10.729857 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:10.729867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:10.730651 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:10.730656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:10.730790 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:10.731355 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:10.731372 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:10.734110 (+ 2738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":230,"threads_started":1}
W20260929 20:54:10.734586 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:10.726548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49643 (local address 127.0.97.62:36703)
0929 20:54:10.726696 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:10.726700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:10.727117 (+ 417us) server_negotiation.cc:407] Connection header received
0929 20:54:10.727246 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:10.727250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:10.727311 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:10.727399 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:10.728108 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:10.728850 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:10.730002 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:10.730449 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:10.730553 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:10.734362 (+ 3809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49643: BlockingRecv error: recv got EOF from 127.0.97.3:49643 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 20:54:10 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:10 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:10.876683 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:10.869214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.1:36327)
0929 20:54:10.869666 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:10.869685 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:10.869792 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:10.870069 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:10.870072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:10.870421 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:10.870589 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:10.870598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:10.871553 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:10.871556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:10.872457 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:10.872468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:10.873085 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:10.873089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:10.873208 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:10.873757 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:10.873772 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:10.876435 (+ 2663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":253,"threads_started":1}
W20260929 20:54:10.876868 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:10.869295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36327 (local address 127.0.97.62:36703)
0929 20:54:10.869511 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:10.869516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:10.869786 (+ 270us) server_negotiation.cc:407] Connection header received
0929 20:54:10.869882 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:10.869886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:10.869962 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:10.870043 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:10.870708 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:10.871444 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:10.872577 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:10.872974 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:10.873062 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:10.876634 (+ 3572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36327: BlockingRecv error: recv got EOF from 127.0.97.1:36327 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 29 20:54:11 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:11 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:11.322049 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:11.313860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.2:54309)
0929 20:54:11.314331 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:11.314347 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:11.314484 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:11.314790 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:11.314795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:11.315170 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:11.315353 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:11.315363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:11.316341 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:11.316347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:11.317587 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:11.317602 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:11.318455 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:11.318462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:11.318627 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:11.319320 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:11.319337 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:11.321805 (+ 2468us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":237,"threads_started":1}
W20260929 20:54:11.322153 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:11.313983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54309 (local address 127.0.97.62:36703)
0929 20:54:11.314164 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:11.314170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:11.314429 (+ 259us) server_negotiation.cc:407] Connection header received
0929 20:54:11.314604 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:11.314608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:11.314668 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:11.314744 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:11.315490 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:11.316211 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:11.317712 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:11.318314 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:11.318424 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:11.321980 (+ 3556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54309: BlockingRecv error: recv got EOF from 127.0.97.2:54309 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 20:54:11 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:11 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:11.742909 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:11.735340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.3:56549)
0929 20:54:11.735759 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:11.735776 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:11.735892 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:11.736169 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:11.736172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:11.736504 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:11.736669 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:11.736677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:11.737654 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:11.737659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:11.738629 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:11.738641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:11.739293 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:11.739297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:11.739406 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:11.739929 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:11.739945 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:11.742571 (+ 2626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":212,"threads_started":1}
W20260929 20:54:11.743016 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:11.735382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56549 (local address 127.0.97.62:36703)
0929 20:54:11.735541 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:11.735545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:11.735878 (+ 333us) server_negotiation.cc:407] Connection header received
0929 20:54:11.735994 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:11.735998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:11.736059 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:11.736137 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:11.736797 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:11.737520 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:11.738762 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:11.739179 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:11.739271 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:11.742803 (+ 3532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56549: BlockingRecv error: recv got EOF from 127.0.97.3:56549 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 20:54:11 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:11 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:11.885329 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:11.877666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.1:43737)
0929 20:54:11.878100 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:11.878119 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:11.878262 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:11.878575 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:11.878579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:11.878970 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:11.879149 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:11.879157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:11.880133 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:11.880137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:11.881047 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:11.881057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:11.881689 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:11.881693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:11.881803 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:11.882387 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:11.882404 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:11.884996 (+ 2592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":210,"threads_started":1}
W20260929 20:54:11.885433 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:11.877739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43737 (local address 127.0.97.62:36703)
0929 20:54:11.877936 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:11.877942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:11.878202 (+ 260us) server_negotiation.cc:407] Connection header received
0929 20:54:11.878353 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:11.878357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:11.878429 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:11.878506 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:11.879283 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:11.880005 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:11.881151 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:11.881581 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:11.881669 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:11.885200 (+ 3531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43737: BlockingRecv error: recv got EOF from 127.0.97.1:43737 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 20:54:12 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:54:12 dist-test-slave-m9g1 krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:54:12.334992 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:12.323093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36703 (local address 127.0.97.2:58759)
0929 20:54:12.323587 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:12.323606 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:12.323732 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:12.324008 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:12.324011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:12.324416 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:12.324601 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:12.324609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:12.325544 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:12.325547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:12.326480 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:12.326491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:12.327154 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:12.327159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:12.327297 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:12.331860 (+ 4563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:12.331877 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:12.334729 (+ 2852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":272,"threads_started":1}
W20260929 20:54:12.335083 9334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:12.323196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58759 (local address 127.0.97.62:36703)
0929 20:54:12.323351 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:12.323355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:12.323695 (+ 340us) server_negotiation.cc:407] Connection header received
0929 20:54:12.323840 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:12.323844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:12.323904 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:12.323992 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:12.324718 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:12.325438 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:12.326581 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:12.327007 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:12.327097 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:12.334896 (+ 7799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58759: BlockingRecv error: recv got EOF from 127.0.97.2:58759 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 20:54:12.590142 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 9185
I20260929 20:54:12.601111 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:12.591132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49416 (local address 127.0.97.1:35323)
0929 20:54:12.591619 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:12.591626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:12.591653 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:54:12.591713 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:12.591719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:12.591936 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:12.592084 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:12.593327 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:12.594430 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:12.595601 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:12.596118 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:12.596208 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:12.598388 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:12.598397 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:12.598411 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:54:12.598444 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:12.600667 (+ 2223us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 20:54:12.600898 (+ 231us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49416: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":236,"threads_started":1}
W20260929 20:54:12.601239 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 9185: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:35323: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 20:54:12.601305 9645 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49416: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 20:54:12.601456 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 9185
I20260929 20:54:12.615391 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 9336
I20260929 20:54:12.626518 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:12.616252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36242 (local address 127.0.97.2:38105)
0929 20:54:12.616774 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:12.616781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:12.616812 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:54:12.616870 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:12.616877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:12.617095 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:12.617259 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:12.618526 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:12.619570 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:12.620742 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:12.621294 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:12.621390 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:12.623826 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:12.623837 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:12.623852 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:54:12.623889 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:12.626028 (+ 2139us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 20:54:12.626296 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36242: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":230,"threads_started":1}
W20260929 20:54:12.626607 9646 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36242: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 20:54:12.626663 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 9336: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:38105: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 20:54:12.626792 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 9336
I20260929 20:54:12.639509 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 9467
I20260929 20:54:12.650054 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:12.640345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37080 (local address 127.0.97.3:43999)
0929 20:54:12.640887 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:12.640894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:12.640919 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:54:12.640976 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:12.640982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:12.641191 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:12.641334 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:12.642493 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:12.643502 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:12.644577 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:12.645101 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:12.645191 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:12.647353 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:12.647360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:12.647373 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:54:12.647411 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:12.649578 (+ 2167us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 20:54:12.649808 (+ 230us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37080: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":256,"threads_started":1}
W20260929 20:54:12.650146 9647 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37080: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 20:54:12.650132 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 9467: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:43999: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 20:54:12.650324 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 9467
I20260929 20:54:12.662726 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 9116
I20260929 20:54:12.663839 9178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:54:12.774546 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 9116
2026-09-29T20:54:12Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20598 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T20:54:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:54:12Z Disabled control of system clock
I20260929 20:54:12.840632 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35989
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 20:54:13.145967 9652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:54:13.146310 9652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:54:13.146400 9652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:54:13.153335 9652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:54:13.153430 9652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:54:13.153472 9652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:54:13.153525 9652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:54:13.153568 9652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:54:13.162441 9652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35989
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:54:13.163628 9652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:54:13.165318 9652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:54:13.176551 9660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:13.176613 9657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:13.176613 9658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:54:13.177098 9652 server_base.cc:1034] running on GCE node
I20260929 20:54:13.177863 9652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:54:13.179189 9652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:54:13.180380 9652 hybrid_clock.cc:648] HybridClock initialized: now 1790715253180339 us; error 39 us; skew 500 ppm
I20260929 20:54:13.180686 9652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:54:13.182765 9652 webserver.cc:460] Webserver started at http://127.0.97.62:44945/ using document root <none> and password file <none>
I20260929 20:54:13.183387 9652 fs_manager.cc:359] Metadata directory not provided
I20260929 20:54:13.183480 9652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:54:13.183775 9652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:54:13.186494 9652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "8e2bede13e2c4f9da122df255b8abbab"
format_stamp: "Formatted at 2026-09-29 20:54:13 on dist-test-slave-m9g1"
server_key: "10af494f93cc1d9308ad06adb86933d9"
server_key_iv: "cea5c28799cba3b692b3275d304b8812"
server_key_version: "encryptionkey@0"
I20260929 20:54:13.187254 9652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "8e2bede13e2c4f9da122df255b8abbab"
format_stamp: "Formatted at 2026-09-29 20:54:13 on dist-test-slave-m9g1"
server_key: "10af494f93cc1d9308ad06adb86933d9"
server_key_iv: "cea5c28799cba3b692b3275d304b8812"
server_key_version: "encryptionkey@0"
I20260929 20:54:13.192052 9652 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 20:54:13.195355 9666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:13.196593 9652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:54:13.196753 9652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "8e2bede13e2c4f9da122df255b8abbab"
format_stamp: "Formatted at 2026-09-29 20:54:13 on dist-test-slave-m9g1"
server_key: "10af494f93cc1d9308ad06adb86933d9"
server_key_iv: "cea5c28799cba3b692b3275d304b8812"
server_key_version: "encryptionkey@0"
I20260929 20:54:13.196965 9652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:54:13.245991 9652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:54:13.246869 9652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:54:13.247126 9652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:54:13.263456 9652 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35989
I20260929 20:54:13.263463 9717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35989 every 8 connection(s)
I20260929 20:54:13.264741 9652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:54:13.268246 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 9652
I20260929 20:54:13.268424 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:54:13.268585 9718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:54:13.269203 388 external_mini_cluster.cc:1428] Setting key 3a856365b9e637b922872c87924319f3
I20260929 20:54:13.281214 9718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab: Bootstrap starting.
I20260929 20:54:13.282872 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:13.271205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.0.97.62:35989)
0929 20:54:13.272166 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:13.272176 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:13.272238 (+ 62us) server_negotiation.cc:407] Connection header received
0929 20:54:13.274406 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:13.274427 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:13.274890 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:13.275191 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:13.275686 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:13.276843 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:13.278660 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:13.279346 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:13.279474 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:13.280599 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:13.280607 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:13.280621 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:54:13.281159 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:13.281257 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:54:13.281279 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:13.281385 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:13.281587 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 20:54:13.281894 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":330,"threads_started":1}
I20260929 20:54:13.284447 9718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab: Neither blocks nor log segments found. Creating new log.
I20260929 20:54:13.285630 9718 log.cc:824] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab: Log is configured to *not* fsync() on all Append() calls
I20260929 20:54:13.288064 9718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab: No bootstrap required, opened a new log
I20260929 20:54:13.294898 9718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2bede13e2c4f9da122df255b8abbab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35989 } }
I20260929 20:54:13.295244 9718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:54:13.295329 9718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e2bede13e2c4f9da122df255b8abbab, State: Initialized, Role: FOLLOWER
I20260929 20:54:13.295845 9718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2bede13e2c4f9da122df255b8abbab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35989 } }
I20260929 20:54:13.296038 9718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:54:13.296149 9718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:54:13.296293 9718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:54:13.298527 9718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2bede13e2c4f9da122df255b8abbab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35989 } }
I20260929 20:54:13.298962 9718 leader_election.cc:302] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e2bede13e2c4f9da122df255b8abbab; no voters:
I20260929 20:54:13.299451 9718 leader_election.cc:288] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:54:13.299566 9723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:54:13.300258 9723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab [term 1 LEADER]: Becoming Leader. State: Replica: 8e2bede13e2c4f9da122df255b8abbab, State: Running, Role: LEADER
I20260929 20:54:13.300817 9723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2bede13e2c4f9da122df255b8abbab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35989 } }
I20260929 20:54:13.301390 9718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:54:13.303422 9725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e2bede13e2c4f9da122df255b8abbab. Latest consensus state: current_term: 1 leader_uuid: "8e2bede13e2c4f9da122df255b8abbab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2bede13e2c4f9da122df255b8abbab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35989 } } }
I20260929 20:54:13.303771 9725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab [sys.catalog]: This master's current role is: LEADER
I20260929 20:54:13.304370 9724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e2bede13e2c4f9da122df255b8abbab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2bede13e2c4f9da122df255b8abbab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35989 } } }
I20260929 20:54:13.304688 9724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab [sys.catalog]: This master's current role is: LEADER
I20260929 20:54:13.307060 9730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:54:13.311758 9730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:54:13.320904 9730 catalog_manager.cc:1347] Generated new cluster ID: 2e94d5b05e1c484e849a23da4b23488c
I20260929 20:54:13.321015 9730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:54:13.345979 9730 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:54:13.347275 9730 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:54:13.354540 9730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e2bede13e2c4f9da122df255b8abbab: Generated new TSK 0
I20260929 20:54:13.355288 9730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:54:13.370029 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35989
--builtin_ntp_servers=127.0.97.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 20:54:13.678980 9742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:54:13.679239 9742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:54:13.679356 9742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:54:13.686141 9742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:54:13.686250 9742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:54:13.686385 9742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:54:13.695143 9742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:54:13.696357 9742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:54:13.698042 9742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:54:13.710500 9748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:13.710582 9747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:13.711030 9750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:54:13.711036 9742 server_base.cc:1034] running on GCE node
I20260929 20:54:13.711712 9742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:54:13.712580 9742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:54:13.713774 9742 hybrid_clock.cc:648] HybridClock initialized: now 1790715253713719 us; error 50 us; skew 500 ppm
I20260929 20:54:13.714067 9742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:54:13.716228 9742 webserver.cc:460] Webserver started at http://127.0.97.1:35993/ using document root <none> and password file <none>
I20260929 20:54:13.716878 9742 fs_manager.cc:359] Metadata directory not provided
I20260929 20:54:13.716977 9742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:54:13.717285 9742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:54:13.720037 9742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "21fcfd498a544dca989c5217a4820c5d"
format_stamp: "Formatted at 2026-09-29 20:54:13 on dist-test-slave-m9g1"
server_key: "51eceeb4a5a729f3821f6dd71cd251bf"
server_key_iv: "7ba91b1902bd688e753e022aa1356a24"
server_key_version: "encryptionkey@0"
I20260929 20:54:13.720789 9742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "21fcfd498a544dca989c5217a4820c5d"
format_stamp: "Formatted at 2026-09-29 20:54:13 on dist-test-slave-m9g1"
server_key: "51eceeb4a5a729f3821f6dd71cd251bf"
server_key_iv: "7ba91b1902bd688e753e022aa1356a24"
server_key_version: "encryptionkey@0"
I20260929 20:54:13.725539 9742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:54:13.728740 9756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:13.729959 9742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:54:13.730115 9742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "21fcfd498a544dca989c5217a4820c5d"
format_stamp: "Formatted at 2026-09-29 20:54:13 on dist-test-slave-m9g1"
server_key: "51eceeb4a5a729f3821f6dd71cd251bf"
server_key_iv: "7ba91b1902bd688e753e022aa1356a24"
server_key_version: "encryptionkey@0"
I20260929 20:54:13.730335 9742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:54:13.757228 9742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:54:13.758015 9742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:54:13.758288 9742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:54:13.759141 9742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:54:13.760492 9742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:54:13.760581 9742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:13.760682 9742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:54:13.760752 9742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:13.788760 9742 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44375
I20260929 20:54:13.788810 9868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44375 every 8 connection(s)
I20260929 20:54:13.789996 9742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:54:13.791750 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 9742
I20260929 20:54:13.791901 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:54:13.792536 388 external_mini_cluster.cc:1428] Setting key 7bc6c49e8f8d03d9a83547fd36f87b95
I20260929 20:54:13.795122 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35989
--builtin_ntp_servers=127.0.97.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 20:54:13.802175 9871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:13.792554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49721 (local address 127.0.97.62:35989)
0929 20:54:13.793008 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:13.793014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:13.795369 (+ 2355us) server_negotiation.cc:407] Connection header received
0929 20:54:13.796146 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:13.796150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:13.796231 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:13.796319 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:13.797357 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:13.798066 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:13.799588 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:13.800055 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:13.800179 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:13.801500 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:13.801505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:13.801508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:54:13.801524 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:13.801569 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:54:13.801581 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:13.801648 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:13.801866 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 20:54:13.801935 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":215,"threads_started":1}
I20260929 20:54:13.803460 9872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:13.793173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35989 (local address 127.0.97.1:49721)
0929 20:54:13.794256 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:13.794307 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:13.795900 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:13.796550 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:13.796558 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:13.796653 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:13.797204 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:13.797221 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:13.798241 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:13.798245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:13.799458 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:13.799469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:13.800256 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:13.800262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:13.800495 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:13.801155 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:54:13.801159 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:13.801207 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:54:13.801300 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:54:13.801664 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:54:13.801672 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:54:13.801680 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:54:13.801859 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 20:54:13.802249 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":300,"threads_started":1}
I20260929 20:54:13.804867 9869 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35989
I20260929 20:54:13.805141 9869 heartbeater.cc:464] Registering TS with master...
I20260929 20:54:13.805815 9869 heartbeater.cc:511] Master 127.0.97.62:35989 requested a full tablet report, sending...
I20260929 20:54:13.808130 9683 ts_manager.cc:194] Registered new tserver with Master: 21fcfd498a544dca989c5217a4820c5d (127.0.97.1:44375)
I20260929 20:54:13.810101 9683 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:49721
W20260929 20:54:14.099292 9873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:54:14.099566 9873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:54:14.099658 9873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:54:14.106870 9873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:54:14.106964 9873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:54:14.107069 9873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:54:14.115911 9873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:54:14.117098 9873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:54:14.118741 9873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:54:14.130720 9878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:14.130705 9879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:14.130868 9881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:54:14.131456 9873 server_base.cc:1034] running on GCE node
I20260929 20:54:14.132012 9873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:54:14.132903 9873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:54:14.134131 9873 hybrid_clock.cc:648] HybridClock initialized: now 1790715254134074 us; error 45 us; skew 500 ppm
I20260929 20:54:14.134474 9873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:54:14.136539 9873 webserver.cc:460] Webserver started at http://127.0.97.2:46401/ using document root <none> and password file <none>
I20260929 20:54:14.137173 9873 fs_manager.cc:359] Metadata directory not provided
I20260929 20:54:14.137265 9873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:54:14.137562 9873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:54:14.140277 9873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "a3765a442dfb42e8bf66d0c8d48d17ad"
format_stamp: "Formatted at 2026-09-29 20:54:14 on dist-test-slave-m9g1"
server_key: "8a532891699e83b546e796f764febd6f"
server_key_iv: "0afca659a3f8ba1b8372080cca72ca4d"
server_key_version: "encryptionkey@0"
I20260929 20:54:14.141008 9873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a3765a442dfb42e8bf66d0c8d48d17ad"
format_stamp: "Formatted at 2026-09-29 20:54:14 on dist-test-slave-m9g1"
server_key: "8a532891699e83b546e796f764febd6f"
server_key_iv: "0afca659a3f8ba1b8372080cca72ca4d"
server_key_version: "encryptionkey@0"
I20260929 20:54:14.145684 9873 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:54:14.148761 9887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:14.149942 9873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:54:14.150094 9873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "a3765a442dfb42e8bf66d0c8d48d17ad"
format_stamp: "Formatted at 2026-09-29 20:54:14 on dist-test-slave-m9g1"
server_key: "8a532891699e83b546e796f764febd6f"
server_key_iv: "0afca659a3f8ba1b8372080cca72ca4d"
server_key_version: "encryptionkey@0"
I20260929 20:54:14.150337 9873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:54:14.196238 9873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:54:14.197104 9873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:54:14.197355 9873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:54:14.198274 9873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:54:14.199700 9873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:54:14.199785 9873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:14.199889 9873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:54:14.199982 9873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:14.228070 9873 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42729
I20260929 20:54:14.228112 9999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42729 every 8 connection(s)
I20260929 20:54:14.229324 9873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:54:14.235642 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 9873
I20260929 20:54:14.235807 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:54:14.236420 388 external_mini_cluster.cc:1428] Setting key a07902bb43b4a99f6ccdbcdd4ed49745
I20260929 20:54:14.238935 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35989
--builtin_ntp_servers=127.0.97.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 20:54:14.241930 9871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:14.231691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35767 (local address 127.0.97.62:35989)
0929 20:54:14.231882 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:14.231886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:14.233235 (+ 1349us) server_negotiation.cc:407] Connection header received
0929 20:54:14.234996 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:14.235000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:14.235063 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:14.235139 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:14.236461 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:14.237230 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:14.239389 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:14.239990 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:14.240113 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:14.241257 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:14.241263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:14.241267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:54:14.241284 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:14.241327 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:54:14.241340 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:14.241420 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:14.241573 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 20:54:14.241657 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:54:14.242872 10002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:14.232220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35989 (local address 127.0.97.2:35767)
0929 20:54:14.233070 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:14.233109 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:14.234745 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:14.235388 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:14.235399 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:14.235516 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:14.236292 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:14.236315 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:14.237879 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:14.237884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:14.239262 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:14.239273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:14.240165 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:14.240170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:14.240356 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:14.240982 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:54:14.240985 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:14.241022 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:54:14.241095 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:54:14.241416 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:54:14.241421 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:54:14.241427 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 20:54:14.241561 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 20:54:14.241834 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":258,"threads_started":1}
I20260929 20:54:14.244144 10000 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35989
I20260929 20:54:14.244397 10000 heartbeater.cc:464] Registering TS with master...
I20260929 20:54:14.245075 10000 heartbeater.cc:511] Master 127.0.97.62:35989 requested a full tablet report, sending...
I20260929 20:54:14.246384 9683 ts_manager.cc:194] Registered new tserver with Master: a3765a442dfb42e8bf66d0c8d48d17ad (127.0.97.2:42729)
I20260929 20:54:14.247344 9683 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:35767
W20260929 20:54:14.549489 10003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:54:14.549772 10003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:54:14.549852 10003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:54:14.557081 10003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:54:14.557176 10003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:54:14.557277 10003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:54:14.566246 10003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:54:14.567448 10003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:54:14.569159 10003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:54:14.582762 10008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:14.582813 10009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:14.582989 10011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:54:14.583436 10003 server_base.cc:1034] running on GCE node
I20260929 20:54:14.583954 10003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:54:14.584801 10003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:54:14.585996 10003 hybrid_clock.cc:648] HybridClock initialized: now 1790715254585946 us; error 44 us; skew 500 ppm
I20260929 20:54:14.586313 10003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:54:14.588404 10003 webserver.cc:460] Webserver started at http://127.0.97.3:44731/ using document root <none> and password file <none>
I20260929 20:54:14.589020 10003 fs_manager.cc:359] Metadata directory not provided
I20260929 20:54:14.589105 10003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:54:14.589391 10003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:54:14.592067 10003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "18c7990bb5c040349493e804b2d453ad"
format_stamp: "Formatted at 2026-09-29 20:54:14 on dist-test-slave-m9g1"
server_key: "03e6c681d53236fe0b2fdae4ef74825e"
server_key_iv: "51528fc55a8fdb95ef4268a7087f5049"
server_key_version: "encryptionkey@0"
I20260929 20:54:14.592783 10003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "18c7990bb5c040349493e804b2d453ad"
format_stamp: "Formatted at 2026-09-29 20:54:14 on dist-test-slave-m9g1"
server_key: "03e6c681d53236fe0b2fdae4ef74825e"
server_key_iv: "51528fc55a8fdb95ef4268a7087f5049"
server_key_version: "encryptionkey@0"
I20260929 20:54:14.597504 10003 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:54:14.600999 10017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:14.602396 10003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:54:14.602540 10003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "18c7990bb5c040349493e804b2d453ad"
format_stamp: "Formatted at 2026-09-29 20:54:14 on dist-test-slave-m9g1"
server_key: "03e6c681d53236fe0b2fdae4ef74825e"
server_key_iv: "51528fc55a8fdb95ef4268a7087f5049"
server_key_version: "encryptionkey@0"
I20260929 20:54:14.602732 10003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:54:14.622782 10003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:54:14.623557 10003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:54:14.623809 10003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:54:14.624702 10003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:54:14.626103 10003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:54:14.626219 10003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:14.626327 10003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:54:14.626399 10003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:14.654760 10003 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32959
I20260929 20:54:14.654835 10129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32959 every 8 connection(s)
I20260929 20:54:14.655917 10003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:54:14.659360 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 10003
I20260929 20:54:14.659514 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:54:14.660053 388 external_mini_cluster.cc:1428] Setting key 29ccecabff181cd42105f0cec55ea874
I20260929 20:54:14.668017 9871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:14.658161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47207 (local address 127.0.97.62:35989)
0929 20:54:14.658353 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:14.658357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:14.659999 (+ 1642us) server_negotiation.cc:407] Connection header received
0929 20:54:14.662139 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:14.662144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:14.662244 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:14.662344 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:14.663521 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:14.664226 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:14.665716 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:14.666119 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:14.666257 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:14.667404 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:14.667409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:14.667413 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:54:14.667429 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:14.667459 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:54:14.667471 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:14.667557 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:14.667721 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 20:54:14.667791 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:54:14.668941 10132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:14.658670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35989 (local address 127.0.97.3:47207)
0929 20:54:14.659833 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:14.659879 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:14.661890 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:14.662596 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:14.662608 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:14.662722 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:14.663344 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:14.663361 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:14.664353 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:14.664357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:14.665584 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:14.665595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:14.666324 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:14.666329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:14.666514 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:14.667095 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:54:14.667097 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:14.667137 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:54:14.667211 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:54:14.667559 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:54:14.667565 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:54:14.667571 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 20:54:14.667701 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 20:54:14.667970 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":261,"threads_started":1}
I20260929 20:54:14.670229 10130 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35989
I20260929 20:54:14.670496 10130 heartbeater.cc:464] Registering TS with master...
I20260929 20:54:14.671212 10130 heartbeater.cc:511] Master 127.0.97.62:35989 requested a full tablet report, sending...
I20260929 20:54:14.672436 9683 ts_manager.cc:194] Registered new tserver with Master: 18c7990bb5c040349493e804b2d453ad (127.0.97.3:32959)
I20260929 20:54:14.673171 9683 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:47207
I20260929 20:54:14.674983 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 20:54:14.680562 9871 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:14.679282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43630 (local address 127.0.97.62:35989)
0929 20:54:14.679468 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:14.679475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:14.679783 (+ 308us) server_negotiation.cc:407] Connection header received
0929 20:54:14.679903 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:14.679907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:14.679975 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:14.680051 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:14.680359 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43630: BlockingRecv error: recv got EOF from 127.0.0.1:43630 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:54:14.681231 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 9742
I20260929 20:54:14.688004 10139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:14.682066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45386 (local address 127.0.97.1:44375)
0929 20:54:14.682543 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:14.682550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:14.682575 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:54:14.682624 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:14.682629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:14.682808 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:14.682918 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:14.683230 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:14.684189 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:14.685269 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:14.685776 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:14.685864 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:14.686753 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:14.686761 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:14.686773 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:54:14.687334 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:14.687380 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:54:14.687398 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:14.687491 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:14.687567 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 20:54:14.687670 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":248,"threads_started":1}
I20260929 20:54:14.689105 9864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:54:14.768936 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 9742
I20260929 20:54:14.781931 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 9873
I20260929 20:54:14.789124 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:14.782766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54042 (local address 127.0.97.2:42729)
0929 20:54:14.783346 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:14.783353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:14.783381 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:54:14.783444 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:14.783450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:14.783672 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:14.783798 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:14.784139 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:14.785131 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:14.786351 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:14.786888 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:14.786977 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:14.787866 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:14.787875 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:14.787889 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:54:14.788483 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:14.788538 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:54:14.788563 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:14.788661 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:14.788733 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 20:54:14.788841 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":316,"threads_started":1}
I20260929 20:54:14.790361 9995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:54:14.869391 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 9873
I20260929 20:54:14.881764 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 10003
I20260929 20:54:14.889350 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:14.882686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.0.97.3:32959)
0929 20:54:14.883252 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:14.883259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:14.883289 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:54:14.883344 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:14.883350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:14.883566 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:14.883705 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:14.884129 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:14.885131 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:14.886388 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:14.886927 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:14.887037 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:14.888011 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:14.888019 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:14.888033 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:54:14.888597 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:14.888642 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:54:14.888661 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:14.888756 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:14.888898 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 20:54:14.889021 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":284,"threads_started":1}
I20260929 20:54:14.890588 10125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:54:14.971022 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 10003
I20260929 20:54:14.983824 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 9652
I20260929 20:54:14.984977 9713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:54:15.085562 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 9652
2026-09-29T20:54:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2306 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T20:54:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:54:15Z Disabled control of system clock
I20260929 20:54:15.148788 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32799
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 20:54:15.457510 10149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:54:15.457779 10149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:54:15.457861 10149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:54:15.464738 10149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:54:15.464849 10149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:54:15.464884 10149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:54:15.464944 10149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:54:15.464986 10149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:54:15.473945 10149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32799
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:54:15.475232 10149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:54:15.476980 10149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:54:15.487798 10154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:15.487728 10157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:15.487766 10155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:54:15.488515 10149 server_base.cc:1034] running on GCE node
I20260929 20:54:15.489243 10149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:54:15.490638 10149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:54:15.491832 10149 hybrid_clock.cc:648] HybridClock initialized: now 1790715255491787 us; error 42 us; skew 500 ppm
I20260929 20:54:15.492152 10149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:54:15.494253 10149 webserver.cc:460] Webserver started at http://127.0.97.62:42351/ using document root <none> and password file <none>
I20260929 20:54:15.494920 10149 fs_manager.cc:359] Metadata directory not provided
I20260929 20:54:15.495014 10149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:54:15.495296 10149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:54:15.498145 10149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "55926a781c1e42e9beb8c3c7e9c40fd5"
format_stamp: "Formatted at 2026-09-29 20:54:15 on dist-test-slave-m9g1"
server_key: "b2ea00461cad7799c993edc76dbcd2ba"
server_key_iv: "214e159a275381768cf2ac0fc0dfd1bb"
server_key_version: "encryptionkey@0"
I20260929 20:54:15.498956 10149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "55926a781c1e42e9beb8c3c7e9c40fd5"
format_stamp: "Formatted at 2026-09-29 20:54:15 on dist-test-slave-m9g1"
server_key: "b2ea00461cad7799c993edc76dbcd2ba"
server_key_iv: "214e159a275381768cf2ac0fc0dfd1bb"
server_key_version: "encryptionkey@0"
I20260929 20:54:15.503865 10149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:54:15.507078 10163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:15.508333 10149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:54:15.508481 10149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "55926a781c1e42e9beb8c3c7e9c40fd5"
format_stamp: "Formatted at 2026-09-29 20:54:15 on dist-test-slave-m9g1"
server_key: "b2ea00461cad7799c993edc76dbcd2ba"
server_key_iv: "214e159a275381768cf2ac0fc0dfd1bb"
server_key_version: "encryptionkey@0"
I20260929 20:54:15.508672 10149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:54:15.514106 10149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:54:15.514844 10149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:54:15.515100 10149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:54:15.531320 10149 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32799
I20260929 20:54:15.531351 10214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32799 every 8 connection(s)
I20260929 20:54:15.532554 10149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:54:15.536180 10215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:54:15.538071 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 10149
I20260929 20:54:15.538280 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:54:15.538862 388 external_mini_cluster.cc:1428] Setting key 98c02a6c36875db3e3b9c7ed4796f890
I20260929 20:54:15.546593 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:15.540729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36048 (local address 127.0.97.62:32799)
0929 20:54:15.541682 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:15.541693 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:15.541730 (+ 37us) server_negotiation.cc:407] Connection header received
0929 20:54:15.543292 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:15.543313 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:15.543693 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:15.544050 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:15.545052 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:15.545057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:15.545069 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:54:15.545086 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:15.545146 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:54:15.545161 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:15.545370 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:15.545491 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 20:54:15.545710 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":360,"threads_started":1}
I20260929 20:54:15.547180 10215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5: Bootstrap starting.
I20260929 20:54:15.550513 10215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5: Neither blocks nor log segments found. Creating new log.
I20260929 20:54:15.551527 10215 log.cc:824] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5: Log is configured to *not* fsync() on all Append() calls
I20260929 20:54:15.553898 10215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5: No bootstrap required, opened a new log
I20260929 20:54:15.560699 10215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55926a781c1e42e9beb8c3c7e9c40fd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32799 } }
I20260929 20:54:15.561048 10215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:54:15.561157 10215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55926a781c1e42e9beb8c3c7e9c40fd5, State: Initialized, Role: FOLLOWER
I20260929 20:54:15.561759 10215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55926a781c1e42e9beb8c3c7e9c40fd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32799 } }
I20260929 20:54:15.561971 10215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:54:15.562091 10215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:54:15.562247 10215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:54:15.564270 10215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55926a781c1e42e9beb8c3c7e9c40fd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32799 } }
I20260929 20:54:15.564728 10215 leader_election.cc:302] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55926a781c1e42e9beb8c3c7e9c40fd5; no voters:
I20260929 20:54:15.565231 10215 leader_election.cc:288] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:54:15.565358 10220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:54:15.565972 10220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5 [term 1 LEADER]: Becoming Leader. State: Replica: 55926a781c1e42e9beb8c3c7e9c40fd5, State: Running, Role: LEADER
I20260929 20:54:15.566481 10220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55926a781c1e42e9beb8c3c7e9c40fd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32799 } }
I20260929 20:54:15.567157 10215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:54:15.568943 10221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55926a781c1e42e9beb8c3c7e9c40fd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55926a781c1e42e9beb8c3c7e9c40fd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32799 } } }
I20260929 20:54:15.569325 10221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5 [sys.catalog]: This master's current role is: LEADER
I20260929 20:54:15.574285 10222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55926a781c1e42e9beb8c3c7e9c40fd5. Latest consensus state: current_term: 1 leader_uuid: "55926a781c1e42e9beb8c3c7e9c40fd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55926a781c1e42e9beb8c3c7e9c40fd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32799 } } }
I20260929 20:54:15.574630 10222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5 [sys.catalog]: This master's current role is: LEADER
I20260929 20:54:15.575838 10227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:54:15.581341 10227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:54:15.589175 10227 catalog_manager.cc:1347] Generated new cluster ID: 999d287386314f7b90ecf528c1074535
I20260929 20:54:15.589280 10227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:54:15.603696 10227 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:54:15.604104 10227 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:54:15.610488 10227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55926a781c1e42e9beb8c3c7e9c40fd5: Generated new TSK 0
I20260929 20:54:15.611275 10227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:54:15.620823 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32799
--builtin_ntp_servers=127.0.97.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 20:54:15.959016 10239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:54:15.959267 10239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:54:15.959400 10239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:54:15.966485 10239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:54:15.966581 10239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:54:15.966722 10239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:54:15.976331 10239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:54:15.977561 10239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:54:15.979255 10239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:54:15.991241 10244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:15.991218 10247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:15.991302 10245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:54:15.995575 10239 server_base.cc:1034] running on GCE node
I20260929 20:54:15.996105 10239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:54:15.996946 10239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:54:15.998135 10239 hybrid_clock.cc:648] HybridClock initialized: now 1790715255998089 us; error 40 us; skew 500 ppm
I20260929 20:54:15.998448 10239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:54:16.000648 10239 webserver.cc:460] Webserver started at http://127.0.97.1:35039/ using document root <none> and password file <none>
I20260929 20:54:16.001314 10239 fs_manager.cc:359] Metadata directory not provided
I20260929 20:54:16.001420 10239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:54:16.001705 10239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:54:16.004590 10239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "c8cd369c76304886856f956e968b37dc"
format_stamp: "Formatted at 2026-09-29 20:54:15 on dist-test-slave-m9g1"
server_key: "28d97327099e397d9aca5c2586dafa9e"
server_key_iv: "ed534c39d374d2932bd3f38ac918a70a"
server_key_version: "encryptionkey@0"
I20260929 20:54:16.005317 10239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c8cd369c76304886856f956e968b37dc"
format_stamp: "Formatted at 2026-09-29 20:54:15 on dist-test-slave-m9g1"
server_key: "28d97327099e397d9aca5c2586dafa9e"
server_key_iv: "ed534c39d374d2932bd3f38ac918a70a"
server_key_version: "encryptionkey@0"
I20260929 20:54:16.010298 10239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 20:54:16.013649 10253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:16.014921 10239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:54:16.015066 10239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "c8cd369c76304886856f956e968b37dc"
format_stamp: "Formatted at 2026-09-29 20:54:15 on dist-test-slave-m9g1"
server_key: "28d97327099e397d9aca5c2586dafa9e"
server_key_iv: "ed534c39d374d2932bd3f38ac918a70a"
server_key_version: "encryptionkey@0"
I20260929 20:54:16.015265 10239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:54:16.021523 10239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:54:16.022296 10239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:54:16.022540 10239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:54:16.023397 10239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:54:16.024700 10239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:54:16.024792 10239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:16.024900 10239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:54:16.024966 10239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:16.054086 10239 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36485
I20260929 20:54:16.054127 10365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36485 every 8 connection(s)
I20260929 20:54:16.055351 10239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:54:16.063091 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:16.057724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37941 (local address 127.0.97.62:32799)
0929 20:54:16.058154 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:16.058160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:16.059440 (+ 1280us) server_negotiation.cc:407] Connection header received
0929 20:54:16.061145 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:16.061149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:16.061198 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:16.061306 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:16.062543 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:16.062547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:16.062550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:54:16.062563 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:16.062596 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:54:16.062606 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:16.062696 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:16.062817 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 20:54:16.062861 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":208,"threads_started":1}
I20260929 20:54:16.063993 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:16.058349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32799 (local address 127.0.97.1:37941)
0929 20:54:16.059284 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:16.059326 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:16.060880 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:16.061527 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:16.061535 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:16.061627 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:16.062287 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:54:16.062289 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:16.062327 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:54:16.062415 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:54:16.062675 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:54:16.062680 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:54:16.062685 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:54:16.062906 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 20:54:16.063113 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":250,"threads_started":1}
I20260929 20:54:16.064306 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 10239
I20260929 20:54:16.064514 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:54:16.065164 388 external_mini_cluster.cc:1428] Setting key 02f3590d23b41357b0e0760facf0d0b4
I20260929 20:54:16.065469 10366 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32799
I20260929 20:54:16.065759 10366 heartbeater.cc:464] Registering TS with master...
I20260929 20:54:16.066434 10366 heartbeater.cc:511] Master 127.0.97.62:32799 requested a full tablet report, sending...
I20260929 20:54:16.068071 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32799
--builtin_ntp_servers=127.0.97.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 20:54:16.068699 10180 ts_manager.cc:194] Registered new tserver with Master: c8cd369c76304886856f956e968b37dc (127.0.97.1:36485)
W20260929 20:54:16.376585 10370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:54:16.376895 10370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:54:16.376981 10370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:54:16.383873 10370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:54:16.383971 10370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:54:16.384076 10370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:54:16.392798 10370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:54:16.393971 10370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:54:16.395684 10370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:54:16.407779 10375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:16.407743 10378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:16.407804 10376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:54:16.408458 10370 server_base.cc:1034] running on GCE node
I20260929 20:54:16.409049 10370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:54:16.409984 10370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:54:16.411186 10370 hybrid_clock.cc:648] HybridClock initialized: now 1790715256411114 us; error 68 us; skew 500 ppm
I20260929 20:54:16.411491 10370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:54:16.413825 10370 webserver.cc:460] Webserver started at http://127.0.97.2:45691/ using document root <none> and password file <none>
I20260929 20:54:16.414553 10370 fs_manager.cc:359] Metadata directory not provided
I20260929 20:54:16.414665 10370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:54:16.414968 10370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:54:16.419404 10370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "19799655674a483b849457e6d67af967"
format_stamp: "Formatted at 2026-09-29 20:54:16 on dist-test-slave-m9g1"
server_key: "fa8862795faf73b3f593e4ebae75f2b7"
server_key_iv: "1254e91d8261b24c605e7e1db9fbcdc1"
server_key_version: "encryptionkey@0"
I20260929 20:54:16.420198 10370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "19799655674a483b849457e6d67af967"
format_stamp: "Formatted at 2026-09-29 20:54:16 on dist-test-slave-m9g1"
server_key: "fa8862795faf73b3f593e4ebae75f2b7"
server_key_iv: "1254e91d8261b24c605e7e1db9fbcdc1"
server_key_version: "encryptionkey@0"
I20260929 20:54:16.425292 10370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 20:54:16.428679 10384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:16.429997 10370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:54:16.430140 10370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "19799655674a483b849457e6d67af967"
format_stamp: "Formatted at 2026-09-29 20:54:16 on dist-test-slave-m9g1"
server_key: "fa8862795faf73b3f593e4ebae75f2b7"
server_key_iv: "1254e91d8261b24c605e7e1db9fbcdc1"
server_key_version: "encryptionkey@0"
I20260929 20:54:16.430384 10370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:54:16.435698 10370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:54:16.436430 10370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:54:16.436659 10370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:54:16.437489 10370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:54:16.438812 10370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:54:16.438910 10370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:16.439011 10370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:54:16.439077 10370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:16.466982 10370 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33291
I20260929 20:54:16.467020 10496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33291 every 8 connection(s)
I20260929 20:54:16.468390 10370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:54:16.475839 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:16.470748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36309 (local address 127.0.97.62:32799)
0929 20:54:16.470895 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:16.470900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:16.472252 (+ 1352us) server_negotiation.cc:407] Connection header received
0929 20:54:16.473845 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:16.473850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:16.473930 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:16.474012 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:16.475195 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:16.475201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:16.475206 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:54:16.475222 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:16.475252 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:54:16.475263 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:16.475371 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:16.475538 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 20:54:16.475600 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 20:54:16.476735 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:16.471252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32799 (local address 127.0.97.2:36309)
0929 20:54:16.472106 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:16.472145 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:16.473601 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:16.474265 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:16.474273 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:16.474363 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:16.474922 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:54:16.474925 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:16.474961 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:54:16.475042 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:54:16.475340 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:54:16.475350 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:54:16.475357 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:54:16.475603 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 20:54:16.475865 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":252,"threads_started":1}
I20260929 20:54:16.477926 10497 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32799
I20260929 20:54:16.478169 10497 heartbeater.cc:464] Registering TS with master...
I20260929 20:54:16.478535 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 10370
I20260929 20:54:16.478708 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:54:16.478869 10497 heartbeater.cc:511] Master 127.0.97.62:32799 requested a full tablet report, sending...
I20260929 20:54:16.479239 388 external_mini_cluster.cc:1428] Setting key d0a2485375855999dfb9cec1845fd89d
I20260929 20:54:16.480345 10180 ts_manager.cc:194] Registered new tserver with Master: 19799655674a483b849457e6d67af967 (127.0.97.2:33291)
I20260929 20:54:16.482105 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32799
--builtin_ntp_servers=127.0.97.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 20:54:16.798753 10500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:54:16.799045 10500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:54:16.799170 10500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:54:16.806547 10500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:54:16.806658 10500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:54:16.806803 10500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:54:16.816330 10500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:54:16.817590 10500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:54:16.819401 10500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:54:16.831302 10506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:16.831748 10508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:16.831787 10505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:54:16.832062 10500 server_base.cc:1034] running on GCE node
I20260929 20:54:16.832605 10500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:54:16.833473 10500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:54:16.834656 10500 hybrid_clock.cc:648] HybridClock initialized: now 1790715256834604 us; error 54 us; skew 500 ppm
I20260929 20:54:16.834949 10500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:54:16.836980 10500 webserver.cc:460] Webserver started at http://127.0.97.3:38727/ using document root <none> and password file <none>
I20260929 20:54:16.837594 10500 fs_manager.cc:359] Metadata directory not provided
I20260929 20:54:16.837687 10500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:54:16.837975 10500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:54:16.840765 10500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "426b11fc4005474f9146d837e8991507"
format_stamp: "Formatted at 2026-09-29 20:54:16 on dist-test-slave-m9g1"
server_key: "20c35e3b53ba1b84c0588b25ab1fd2fc"
server_key_iv: "9cad5677eec0b99426f3a655e8116d5d"
server_key_version: "encryptionkey@0"
I20260929 20:54:16.841492 10500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "426b11fc4005474f9146d837e8991507"
format_stamp: "Formatted at 2026-09-29 20:54:16 on dist-test-slave-m9g1"
server_key: "20c35e3b53ba1b84c0588b25ab1fd2fc"
server_key_iv: "9cad5677eec0b99426f3a655e8116d5d"
server_key_version: "encryptionkey@0"
I20260929 20:54:16.846274 10500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260929 20:54:16.849450 10514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:16.850808 10500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:54:16.850967 10500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "426b11fc4005474f9146d837e8991507"
format_stamp: "Formatted at 2026-09-29 20:54:16 on dist-test-slave-m9g1"
server_key: "20c35e3b53ba1b84c0588b25ab1fd2fc"
server_key_iv: "9cad5677eec0b99426f3a655e8116d5d"
server_key_version: "encryptionkey@0"
I20260929 20:54:16.851171 10500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:54:16.856331 10500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:54:16.857046 10500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:54:16.857272 10500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:54:16.858076 10500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:54:16.859349 10500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:54:16.859442 10500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:16.859542 10500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:54:16.859614 10500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:16.888108 10500 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40045
I20260929 20:54:16.888160 10626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40045 every 8 connection(s)
I20260929 20:54:16.889324 10500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:54:16.890825 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 10500
I20260929 20:54:16.891009 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:54:16.891495 388 external_mini_cluster.cc:1428] Setting key 0ae97411799031aeea72a10f8135f8d6
I20260929 20:54:16.897244 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:16.892005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48693 (local address 127.0.97.62:32799)
0929 20:54:16.892168 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:16.892173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:16.893774 (+ 1601us) server_negotiation.cc:407] Connection header received
0929 20:54:16.895472 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:16.895476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:16.895524 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:16.895600 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:16.896714 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:16.896718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:16.896721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:54:16.896735 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:16.896764 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:54:16.896774 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:16.896863 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:16.896988 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 20:54:16.897031 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:54:16.898084 10629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:16.892585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32799 (local address 127.0.97.3:48693)
0929 20:54:16.893622 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:16.893664 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:16.895236 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:16.895802 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:16.895810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:16.895902 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:16.896456 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:54:16.896458 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:16.896494 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:54:16.896575 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:54:16.896847 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:54:16.896852 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:54:16.896857 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:54:16.897048 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 20:54:16.897235 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":298,"threads_started":1}
I20260929 20:54:16.899245 10627 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32799
I20260929 20:54:16.899498 10627 heartbeater.cc:464] Registering TS with master...
I20260929 20:54:16.900019 10627 heartbeater.cc:511] Master 127.0.97.62:32799 requested a full tablet report, sending...
I20260929 20:54:16.901196 10179 ts_manager.cc:194] Registered new tserver with Master: 426b11fc4005474f9146d837e8991507 (127.0.97.3:40045)
I20260929 20:54:16.906368 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 20:54:16.911958 10368 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:54:16.910557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36064 (local address 127.0.97.62:32799)
0929 20:54:16.910745 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:16.910751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:16.911183 (+ 432us) server_negotiation.cc:407] Connection header received
0929 20:54:16.911314 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:16.911318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:16.911369 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:16.911448 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:16.911697 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36064: BlockingRecv error: recv got EOF from 127.0.0.1:36064 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:54:16.912621 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 10239
I20260929 20:54:16.915861 10636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:16.913517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35830 (local address 127.0.97.1:36485)
0929 20:54:16.913988 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:16.913994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:16.914019 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:54:16.914071 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:16.914076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:16.914316 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:16.914440 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:16.915195 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:16.915201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:16.915212 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:54:16.915228 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:16.915267 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:54:16.915282 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:16.915386 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:16.915564 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 20:54:16.915640 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":224,"threads_started":1}
I20260929 20:54:16.916955 10361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:54:16.994925 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 10239
I20260929 20:54:17.007997 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 10370
I20260929 20:54:17.011217 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:17.008953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60232 (local address 127.0.97.2:33291)
0929 20:54:17.009426 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:17.009433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:17.009461 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:54:17.009520 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:17.009526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:17.009737 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:17.009878 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:17.010686 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:17.010693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:17.010705 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:54:17.010716 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:17.010756 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:54:17.010773 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:17.010874 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:17.010919 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 20:54:17.011001 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":209,"threads_started":1}
I20260929 20:54:17.012509 10492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:54:17.092303 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 10370
I20260929 20:54:17.104650 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 10500
I20260929 20:54:17.108420 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:17.105553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47904 (local address 127.0.97.3:40045)
0929 20:54:17.106154 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:17.106160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:17.106211 (+ 51us) server_negotiation.cc:407] Connection header received
0929 20:54:17.106276 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:17.106283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:17.106502 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:17.106639 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:17.107724 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:17.107731 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:17.107741 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:54:17.107757 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:17.107792 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:54:17.107805 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:17.107955 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:17.108050 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 20:54:17.108144 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":305,"threads_started":1}
I20260929 20:54:17.109586 10622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:54:17.187738 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 10500
I20260929 20:54:17.200101 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 10149
I20260929 20:54:17.201217 10210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:54:17.298157 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 10149
2026-09-29T20:54:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2211 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:54:17 dist-test-slave-m9g1 krb5kdc[10645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:54:17 dist-test-slave-m9g1 krb5kdc[10645](info): set up 2 sockets
Sep 29 20:54:17 dist-test-slave-m9g1 krb5kdc[10645](info): commencing operation
krb5kdc: starting...
W20260929 20:54:18.302698 388 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:54:18 dist-test-slave-m9g1 krb5kdc[10645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:54:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:54:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:54:18.510931 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37939
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:54:18.830763 10658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:54:18.831133 10658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:54:18.831274 10658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:54:18.842311 10658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:54:18.842451 10658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:54:18.842522 10658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:54:18.842600 10658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:54:18.842660 10658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:54:18.852515 10658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:54:18.853785 10658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:54:18.855085 10658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:54:18.865583 10666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:18.865628 10664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:18.865657 10663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:54:18.866391 10658 server_base.cc:1034] running on GCE node
I20260929 20:54:18.867064 10658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:54:18.868292 10658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:54:18.869472 10658 hybrid_clock.cc:648] HybridClock initialized: now 1790715258869434 us; error 31 us; skew 500 ppm
I20260929 20:54:18.869781 10658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:54:18 dist-test-slave-m9g1 krb5kdc[10645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:54:18.872575 10658 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:54:18.873919 10658 webserver.cc:460] Webserver started at http://127.0.97.62:43607/ using document root <none> and password file <none>
I20260929 20:54:18.874590 10658 fs_manager.cc:359] Metadata directory not provided
I20260929 20:54:18.874701 10658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:54:18.874972 10658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:54:18.877728 10658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "198515da9d9444df99c356b45ac7e76b"
format_stamp: "Formatted at 2026-09-29 20:54:18 on dist-test-slave-m9g1"
server_key: "f5dfa24e81c2b822e2b155b0c9ae004d"
server_key_iv: "9e8b950a0108d1445f2bf05cc211c794"
server_key_version: "encryptionkey@0"
I20260929 20:54:18.878450 10658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "198515da9d9444df99c356b45ac7e76b"
format_stamp: "Formatted at 2026-09-29 20:54:18 on dist-test-slave-m9g1"
server_key: "f5dfa24e81c2b822e2b155b0c9ae004d"
server_key_iv: "9e8b950a0108d1445f2bf05cc211c794"
server_key_version: "encryptionkey@0"
I20260929 20:54:18.883145 10658 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:54:18.886315 10673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:18.887569 10658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:54:18.887715 10658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "198515da9d9444df99c356b45ac7e76b"
format_stamp: "Formatted at 2026-09-29 20:54:18 on dist-test-slave-m9g1"
server_key: "f5dfa24e81c2b822e2b155b0c9ae004d"
server_key_iv: "9e8b950a0108d1445f2bf05cc211c794"
server_key_version: "encryptionkey@0"
I20260929 20:54:18.887876 10658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:54:18.914475 10658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:54:18.918591 10658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:54:18.918859 10658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:54:18.935323 10658 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37939
I20260929 20:54:18.935346 10724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37939 every 8 connection(s)
I20260929 20:54:18.936591 10658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:54:18.940133 10725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:54:18.941025 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 10658
I20260929 20:54:18.941243 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:54:18.942034 388 external_mini_cluster.cc:1428] Setting key dff58864abe89208c89b7f9ae3842a67
I20260929 20:54:18.953624 10725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b: Bootstrap starting.
Sep 29 20:54:18 dist-test-slave-m9g1 krb5kdc[10645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:54:18.957343 10725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b: Neither blocks nor log segments found. Creating new log.
I20260929 20:54:18.958447 10725 log.cc:824] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:54:18.961609 10725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b: No bootstrap required, opened a new log
I20260929 20:54:18.964105 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:18.943970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43934 (local address 127.0.97.62:37939)
0929 20:54:18.944652 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:18.944664 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:18.944707 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:54:18.946290 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:18.946312 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:18.946675 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:18.946990 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:18.948631 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:18.949902 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:18.951611 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:18.952280 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:18.952405 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:18.956892 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:18.956901 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:18.956915 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:54:18.956955 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:18.960133 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:54:18.960977 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:54:18.960984 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:54:18.960990 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:54:18.961145 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:54:18.961506 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:54:18.961509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:54:18.961511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:54:18.962037 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:18.962295 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:18.962692 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 20:54:18.963032 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":295,"threads_started":1}
I20260929 20:54:18.969194 10725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198515da9d9444df99c356b45ac7e76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37939 } }
I20260929 20:54:18.969533 10725 raft_consensus.cc:374] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:54:18.969640 10725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198515da9d9444df99c356b45ac7e76b, State: Initialized, Role: FOLLOWER
I20260929 20:54:18.970175 10725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198515da9d9444df99c356b45ac7e76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37939 } }
I20260929 20:54:18.970410 10725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:54:18.970527 10725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:54:18.970672 10725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:54:18.972710 10725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198515da9d9444df99c356b45ac7e76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37939 } }
I20260929 20:54:18.973176 10725 leader_election.cc:302] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 198515da9d9444df99c356b45ac7e76b; no voters:
I20260929 20:54:18.973697 10725 leader_election.cc:288] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:54:18.973815 10730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:54:18.974386 10730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b [term 1 LEADER]: Becoming Leader. State: Replica: 198515da9d9444df99c356b45ac7e76b, State: Running, Role: LEADER
I20260929 20:54:18.974864 10730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198515da9d9444df99c356b45ac7e76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37939 } }
I20260929 20:54:18.975543 10725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:54:18.977591 10732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 198515da9d9444df99c356b45ac7e76b. Latest consensus state: current_term: 1 leader_uuid: "198515da9d9444df99c356b45ac7e76b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198515da9d9444df99c356b45ac7e76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37939 } } }
I20260929 20:54:18.977926 10732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b [sys.catalog]: This master's current role is: LEADER
I20260929 20:54:18.977563 10731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "198515da9d9444df99c356b45ac7e76b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198515da9d9444df99c356b45ac7e76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37939 } } }
I20260929 20:54:18.978127 10731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b [sys.catalog]: This master's current role is: LEADER
I20260929 20:54:18.979555 10735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:54:18.983907 10735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:54:18.991614 10735 catalog_manager.cc:1347] Generated new cluster ID: 513acc82230c400dbdb041b13c8e6c25
I20260929 20:54:18.991703 10735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:54:19.003490 10735 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:54:19.005321 10735 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:54:19.013420 10735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b: Generated new TSK 0
I20260929 20:54:19.014544 10735 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:54:19.077452 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37939
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:54:19.398463 10751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:54:19.398746 10751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:54:19.398846 10751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:54:19.405915 10751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:54:19.406059 10751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:54:19.406201 10751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:54:19.415613 10751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:54:19.416936 10751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:54:19.418286 10751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:54:19.430784 10757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:19.430773 10759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:19.430765 10756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:54:19.431553 10751 server_base.cc:1034] running on GCE node
I20260929 20:54:19.432107 10751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:54:19.432983 10751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:54:19.434175 10751 hybrid_clock.cc:648] HybridClock initialized: now 1790715259434124 us; error 42 us; skew 500 ppm
I20260929 20:54:19.434494 10751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:54:19 dist-test-slave-m9g1 krb5kdc[10645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:54:19.437388 10751 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:54:19.438717 10751 webserver.cc:460] Webserver started at http://127.0.97.1:40239/ using document root <none> and password file <none>
I20260929 20:54:19.439354 10751 fs_manager.cc:359] Metadata directory not provided
I20260929 20:54:19.439465 10751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:54:19.439733 10751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:54:19.442551 10751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "0a9011c9b88e4a2884d491fc4a25a113"
format_stamp: "Formatted at 2026-09-29 20:54:19 on dist-test-slave-m9g1"
server_key: "a37d5ed5c911cf097e3addf8f980e6db"
server_key_iv: "f4333468200336131effa6d87be96a2f"
server_key_version: "encryptionkey@0"
I20260929 20:54:19.443274 10751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0a9011c9b88e4a2884d491fc4a25a113"
format_stamp: "Formatted at 2026-09-29 20:54:19 on dist-test-slave-m9g1"
server_key: "a37d5ed5c911cf097e3addf8f980e6db"
server_key_iv: "f4333468200336131effa6d87be96a2f"
server_key_version: "encryptionkey@0"
I20260929 20:54:19.448168 10751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 20:54:19.451624 10766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:19.452874 10751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:54:19.453013 10751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "0a9011c9b88e4a2884d491fc4a25a113"
format_stamp: "Formatted at 2026-09-29 20:54:19 on dist-test-slave-m9g1"
server_key: "a37d5ed5c911cf097e3addf8f980e6db"
server_key_iv: "f4333468200336131effa6d87be96a2f"
server_key_version: "encryptionkey@0"
I20260929 20:54:19.453174 10751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:54:19.478855 10751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:54:19.482703 10751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:54:19.482965 10751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:54:19.483906 10751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:54:19.485213 10751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:54:19.485287 10751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:19.485405 10751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:54:19.485453 10751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:19.514150 10751 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40707
I20260929 20:54:19.514223 10878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40707 every 8 connection(s)
I20260929 20:54:19.515416 10751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:54:19.518020 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 10751
I20260929 20:54:19.518286 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:54:19.518944 388 external_mini_cluster.cc:1428] Setting key 895774ffe33be5235410f7d2d3aaccf1
Sep 29 20:54:19 dist-test-slave-m9g1 krb5kdc[10645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:54:19.535223 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:19.518095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54477 (local address 127.0.97.62:37939)
0929 20:54:19.518553 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:19.518560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:19.519769 (+ 1209us) server_negotiation.cc:407] Connection header received
0929 20:54:19.521812 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:19.521817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:19.521906 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:19.522000 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:19.523514 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:19.524231 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:19.525774 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:19.526175 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:19.526324 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:19.529765 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:19.529770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:19.529774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:54:19.529803 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:19.532164 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:54:19.532978 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:54:19.532983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:54:19.532986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:54:19.533110 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:54:19.533557 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:54:19.533561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:54:19.533563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:54:19.533843 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:19.533947 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:19.534798 (+ 851us) server_negotiation.cc:299] Negotiation successful
0929 20:54:19.534970 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":230,"threads_started":1}
I20260929 20:54:19.536720 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:19.518784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37939 (local address 127.0.97.1:54477)
0929 20:54:19.519605 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:19.519644 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:19.521554 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:19.522279 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:19.522289 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:19.522786 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:19.523357 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:19.523375 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:19.524371 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:19.524375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:19.525627 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:19.525640 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:19.526405 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:19.526413 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:19.526643 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:19.527263 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:19.527285 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:19.529524 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:54:19.532281 (+ 2757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:54:19.532290 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:54:19.532306 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:54:19.532841 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:54:19.533228 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:54:19.533232 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:54:19.533236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:54:19.533452 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:54:19.533955 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:54:19.533962 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:54:19.534427 (+ 465us) client_negotiation.cc:769] Sending connection context
0929 20:54:19.534734 (+ 307us) client_negotiation.cc:240] Negotiation successful
0929 20:54:19.535274 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":300,"threads_started":1}
I20260929 20:54:19.538260 10879 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37939
I20260929 20:54:19.538587 10879 heartbeater.cc:464] Registering TS with master...
I20260929 20:54:19.539461 10879 heartbeater.cc:511] Master 127.0.97.62:37939 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:54:19.541564 10690 ts_manager.cc:194] Registered new tserver with Master: 0a9011c9b88e4a2884d491fc4a25a113 (127.0.97.1:40707)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 20:54:19.543107 10690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:54:19.565833 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37939
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:54:19.875741 10885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:54:19.876027 10885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:54:19.876152 10885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:54:19.883219 10885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:54:19.883323 10885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:54:19.883490 10885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:54:19.893054 10885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:54:19.894322 10885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:54:19.895581 10885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:54:19.908095 10891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:19.908174 10890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:19.908130 10893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:54:19.908859 10885 server_base.cc:1034] running on GCE node
I20260929 20:54:19.909461 10885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:54:19.910416 10885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:54:19.911602 10885 hybrid_clock.cc:648] HybridClock initialized: now 1790715259911542 us; error 67 us; skew 500 ppm
I20260929 20:54:19.911882 10885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:54:19 dist-test-slave-m9g1 krb5kdc[10645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:54:19.914837 10885 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:54:19.916182 10885 webserver.cc:460] Webserver started at http://127.0.97.2:33121/ using document root <none> and password file <none>
I20260929 20:54:19.916841 10885 fs_manager.cc:359] Metadata directory not provided
I20260929 20:54:19.916934 10885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:54:19.917164 10885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:54:19.920125 10885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "125dc936467a444a91989ece5ffc17c8"
format_stamp: "Formatted at 2026-09-29 20:54:19 on dist-test-slave-m9g1"
server_key: "cc801475223778903c167eb4fba0be5d"
server_key_iv: "700a77043d3c96b3cf52723754b791aa"
server_key_version: "encryptionkey@0"
I20260929 20:54:19.920823 10885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "125dc936467a444a91989ece5ffc17c8"
format_stamp: "Formatted at 2026-09-29 20:54:19 on dist-test-slave-m9g1"
server_key: "cc801475223778903c167eb4fba0be5d"
server_key_iv: "700a77043d3c96b3cf52723754b791aa"
server_key_version: "encryptionkey@0"
I20260929 20:54:19.926028 10885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:54:19.929288 10900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:19.930639 10885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:54:19.930778 10885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "125dc936467a444a91989ece5ffc17c8"
format_stamp: "Formatted at 2026-09-29 20:54:19 on dist-test-slave-m9g1"
server_key: "cc801475223778903c167eb4fba0be5d"
server_key_iv: "700a77043d3c96b3cf52723754b791aa"
server_key_version: "encryptionkey@0"
I20260929 20:54:19.930946 10885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:54:19.955698 10885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:54:19.959901 10885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:54:19.960125 10885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:54:19.961025 10885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:54:19.962329 10885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:54:19.962401 10885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:19.962488 10885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:54:19.962529 10885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:19.990586 10885 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33695
I20260929 20:54:19.990674 11012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33695 every 8 connection(s)
I20260929 20:54:19.991809 10885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:54:19.996449 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 10885
I20260929 20:54:19.996654 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:54:19.997136 388 external_mini_cluster.cc:1428] Setting key e6aa3e5f081d52ba163c549ed18a9477
Sep 29 20:54:19 dist-test-slave-m9g1 krb5kdc[10645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:54:20.013005 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:19.994347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45499 (local address 127.0.97.62:37939)
0929 20:54:19.994516 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:19.994521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:19.995661 (+ 1140us) server_negotiation.cc:407] Connection header received
0929 20:54:19.997983 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:19.997989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:19.998074 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:19.998158 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:19.999906 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.000648 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.002680 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.003085 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.003182 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:20.007133 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:20.007140 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:20.007145 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:54:20.007183 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:20.010274 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:54:20.010945 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:54:20.010950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:54:20.010953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:54:20.011092 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:54:20.011447 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:54:20.011451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:54:20.011454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:54:20.011745 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:20.011873 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:20.012502 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 20:54:20.012711 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:54:20.014101 11015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:19.994776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37939 (local address 127.0.97.2:45499)
0929 20:54:19.995489 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:19.995527 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:19.997714 (+ 2187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:19.998435 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:19.998443 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:19.998932 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:19.999707 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:19.999733 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.000785 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.000791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:20.002545 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:20.002558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.003288 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.003294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:20.003555 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:20.004249 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:20.004278 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:20.006852 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:54:20.010398 (+ 3546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:54:20.010405 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:54:20.010419 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:54:20.010835 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:54:20.011183 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:54:20.011186 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:54:20.011188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:54:20.011355 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:54:20.011840 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:54:20.011846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:54:20.012166 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 20:54:20.012387 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 20:54:20.012783 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":287,"threads_started":1}
I20260929 20:54:20.015506 11013 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37939
I20260929 20:54:20.015832 11013 heartbeater.cc:464] Registering TS with master...
I20260929 20:54:20.016656 11013 heartbeater.cc:511] Master 127.0.97.62:37939 requested a full tablet report, sending...
I20260929 20:54:20.018236 10690 ts_manager.cc:194] Registered new tserver with Master: 125dc936467a444a91989ece5ffc17c8 (127.0.97.2:33695)
I20260929 20:54:20.019014 10690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45499
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:54:20.048900 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37939
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:54:20.354461 11018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:54:20.354729 11018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:54:20.354830 11018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:54:20.361878 11018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:54:20.361974 11018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:54:20.362093 11018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:54:20.371526 11018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:54:20.372728 11018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:54:20.373947 11018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:54:20.386337 11026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:20.386258 11023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:54:20.386853 11024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:54:20.387159 11018 server_base.cc:1034] running on GCE node
I20260929 20:54:20.387776 11018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:54:20.388669 11018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:54:20.389861 11018 hybrid_clock.cc:648] HybridClock initialized: now 1790715260389812 us; error 44 us; skew 500 ppm
I20260929 20:54:20.390163 11018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:54:20 dist-test-slave-m9g1 krb5kdc[10645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:54:20.393169 11018 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:54:20.394557 11018 webserver.cc:460] Webserver started at http://127.0.97.3:44521/ using document root <none> and password file <none>
I20260929 20:54:20.395233 11018 fs_manager.cc:359] Metadata directory not provided
I20260929 20:54:20.395345 11018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:54:20.395612 11018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:54:20.398427 11018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "983547bdc948418f8f5db7ab4eecf5f3"
format_stamp: "Formatted at 2026-09-29 20:54:20 on dist-test-slave-m9g1"
server_key: "69487efbafd75ebb2b607a2014cc5687"
server_key_iv: "b2875d1adf436168b522de579a745082"
server_key_version: "encryptionkey@0"
I20260929 20:54:20.399144 11018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "983547bdc948418f8f5db7ab4eecf5f3"
format_stamp: "Formatted at 2026-09-29 20:54:20 on dist-test-slave-m9g1"
server_key: "69487efbafd75ebb2b607a2014cc5687"
server_key_iv: "b2875d1adf436168b522de579a745082"
server_key_version: "encryptionkey@0"
I20260929 20:54:20.404048 11018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:54:20.407444 11033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:20.408802 11018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:54:20.408953 11018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "983547bdc948418f8f5db7ab4eecf5f3"
format_stamp: "Formatted at 2026-09-29 20:54:20 on dist-test-slave-m9g1"
server_key: "69487efbafd75ebb2b607a2014cc5687"
server_key_iv: "b2875d1adf436168b522de579a745082"
server_key_version: "encryptionkey@0"
I20260929 20:54:20.409150 11018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:54:20.423857 11018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:54:20.427851 11018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:54:20.428114 11018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:54:20.429018 11018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:54:20.430341 11018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:54:20.430414 11018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:20.430526 11018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:54:20.430588 11018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:54:20.458349 11018 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37091
I20260929 20:54:20.458379 11145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37091 every 8 connection(s)
I20260929 20:54:20.459519 11018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:54:20.468683 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 11018
I20260929 20:54:20.468853 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:54:20.469348 388 external_mini_cluster.cc:1428] Setting key 436254d185fd7491014a500a3ee67cad
Sep 29 20:54:20 dist-test-slave-m9g1 krb5kdc[10645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:54:20.477602 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:20.461716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33763 (local address 127.0.97.62:37939)
0929 20:54:20.461913 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:20.461919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:20.463057 (+ 1138us) server_negotiation.cc:407] Connection header received
0929 20:54:20.464720 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:20.464724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:20.464786 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:20.464868 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:20.466281 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.467045 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.468509 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.468950 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.469051 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:20.472780 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:20.472785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:20.472788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:54:20.472819 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:20.475184 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:54:20.475788 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:54:20.475791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:54:20.475794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:54:20.475902 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:54:20.476253 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:54:20.476256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:54:20.476258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:54:20.476497 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:20.476606 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:20.477194 (+ 588us) server_negotiation.cc:299] Negotiation successful
0929 20:54:20.477352 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:54:20.478605 11148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:20.462249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37939 (local address 127.0.97.3:33763)
0929 20:54:20.462909 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:20.462952 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:20.464483 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:20.465049 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:20.465057 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:20.465521 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:54:20.466089 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:20.466106 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.467160 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.467163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:20.468370 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:20.468384 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.469158 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.469165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:20.469435 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:20.470249 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:54:20.470276 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:54:20.472611 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:54:20.475299 (+ 2688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:54:20.475306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:54:20.475318 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:54:20.475698 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:54:20.475976 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:54:20.475979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:54:20.475981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:54:20.476174 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:54:20.476610 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:54:20.476617 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:54:20.476907 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 20:54:20.477117 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 20:54:20.477483 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":256,"threads_started":1}
I20260929 20:54:20.479864 11146 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37939
I20260929 20:54:20.480146 11146 heartbeater.cc:464] Registering TS with master...
I20260929 20:54:20.480834 11146 heartbeater.cc:511] Master 127.0.97.62:37939 requested a full tablet report, sending...
I20260929 20:54:20.482249 10690 ts_manager.cc:194] Registered new tserver with Master: 983547bdc948418f8f5db7ab4eecf5f3 (127.0.97.3:37091)
I20260929 20:54:20.483321 10690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33763
I20260929 20:54:20.484711 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:54:20.500514 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:20.489388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.0.97.62:37939)
0929 20:54:20.489559 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:20.489564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:20.490005 (+ 441us) server_negotiation.cc:407] Connection header received
0929 20:54:20.490130 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:20.490133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:20.490210 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:20.490294 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:20.491608 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.492313 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.493484 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.493874 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.493963 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:20.495963 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:20.495967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:20.495971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:54:20.496001 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:20.498401 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:54:20.498944 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:54:20.498948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:54:20.498950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:54:20.499067 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:54:20.499322 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:54:20.499324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:54:20.499326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:54:20.499632 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:20.499811 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:20.500123 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 20:54:20.500292 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:54:20.504317 10690 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:54:20.510881 10690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:54:20.531925 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:20.523163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33695 (local address 127.0.0.1:48420)
0929 20:54:20.523906 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:20.523920 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:20.524077 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:20.524846 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:20.524852 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:20.524882 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:54:20.525244 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:20.525256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.527313 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.527318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:20.529234 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:20.529248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.530957 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.530963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:20.531147 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:20.531200 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 20:54:20.531323 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 20:54:20.531507 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":42,"thread_start_us":301,"threads_started":1}
I20260929 20:54:20.532137 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:20.522562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45626 (local address 127.0.97.3:37091)
0929 20:54:20.523234 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:20.523242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:20.523997 (+ 755us) server_negotiation.cc:407] Connection header received
0929 20:54:20.524788 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:20.524796 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:20.525023 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:20.525225 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:54:20.526022 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.527531 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.529921 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.531131 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.531252 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:20.531455 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:20.531676 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 20:54:20.531831 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":358,"threads_started":1}
I20260929 20:54:20.532461 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:20.522664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.0.97.2:33695)
0929 20:54:20.523164 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:20.523173 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:20.523996 (+ 823us) server_negotiation.cc:407] Connection header received
0929 20:54:20.524300 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:20.524308 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:20.524610 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:20.525423 (+ 813us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:54:20.525635 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.527174 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.529385 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.530784 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.530906 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:20.531092 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:20.532027 (+ 935us) server_negotiation.cc:299] Negotiation successful
0929 20:54:20.532189 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":741,"threads_started":1}
I20260929 20:54:20.532753 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:20.524318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40707 (local address 127.0.0.1:57600)
0929 20:54:20.524815 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:20.524832 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:20.524947 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:20.525895 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:20.525900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:20.525923 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:54:20.526367 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:20.526379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.528193 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.528198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:20.530393 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:20.530406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.532060 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.532066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:20.532219 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:20.532249 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:54:20.532309 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 20:54:20.532418 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":82,"thread_start_us":239,"threads_started":1}
I20260929 20:54:20.533705 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:20.524629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.0.97.1:40707)
0929 20:54:20.525431 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:20.525437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:20.525457 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:54:20.525520 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:20.525525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:20.525702 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:20.526543 (+ 841us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:54:20.526781 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.528049 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.530538 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.531842 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.532967 (+ 1125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:20.533174 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:20.533235 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 20:54:20.533381 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":541,"threads_started":1}
I20260929 20:54:20.533721 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:20.522262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37091 (local address 127.0.0.1:45626)
0929 20:54:20.523847 (+ 1585us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:20.523890 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:20.524077 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:20.525461 (+ 1384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:20.525465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:20.525489 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:54:20.525820 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:20.525831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.527646 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.527651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:20.529757 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:20.529770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.531326 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.531332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:20.531487 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:20.531517 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:54:20.531620 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 20:54:20.531746 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1334,"mutex_wait_us":26,"thread_start_us":1724,"threads_started":1}
I20260929 20:54:20.536413 10948 tablet_service.cc:1467] Processing CreateTablet for tablet f4dcb9d113a54a73a448018984a8fcb3 (DEFAULT_TABLE table=test-table [id=54b862c8cc854e6b9ebfae28dc9b1df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:54:20.537066 10814 tablet_service.cc:1467] Processing CreateTablet for tablet f4dcb9d113a54a73a448018984a8fcb3 (DEFAULT_TABLE table=test-table [id=54b862c8cc854e6b9ebfae28dc9b1df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:54:20.537880 10948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4dcb9d113a54a73a448018984a8fcb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:54:20.537550 11081 tablet_service.cc:1467] Processing CreateTablet for tablet f4dcb9d113a54a73a448018984a8fcb3 (DEFAULT_TABLE table=test-table [id=54b862c8cc854e6b9ebfae28dc9b1df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:54:20.538249 10814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4dcb9d113a54a73a448018984a8fcb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:54:20.538829 11081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4dcb9d113a54a73a448018984a8fcb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:54:20.545761 10879 heartbeater.cc:503] Master 127.0.97.62:37939 was elected leader, sending a full tablet report...
I20260929 20:54:20.549154 11164 tablet_bootstrap.cc:492] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113: Bootstrap starting.
I20260929 20:54:20.553222 11166 tablet_bootstrap.cc:492] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3: Bootstrap starting.
I20260929 20:54:20.553830 11164 tablet_bootstrap.cc:654] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113: Neither blocks nor log segments found. Creating new log.
I20260929 20:54:20.554584 11165 tablet_bootstrap.cc:492] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8: Bootstrap starting.
I20260929 20:54:20.555099 11164 log.cc:824] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113: Log is configured to *not* fsync() on all Append() calls
I20260929 20:54:20.556376 11166 tablet_bootstrap.cc:654] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3: Neither blocks nor log segments found. Creating new log.
I20260929 20:54:20.557303 11165 tablet_bootstrap.cc:654] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8: Neither blocks nor log segments found. Creating new log.
I20260929 20:54:20.557489 11166 log.cc:824] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3: Log is configured to *not* fsync() on all Append() calls
I20260929 20:54:20.558275 11164 tablet_bootstrap.cc:492] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113: No bootstrap required, opened a new log
I20260929 20:54:20.558369 11165 log.cc:824] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8: Log is configured to *not* fsync() on all Append() calls
I20260929 20:54:20.558519 11164 ts_tablet_manager.cc:1397] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 20:54:20.560601 11166 tablet_bootstrap.cc:492] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3: No bootstrap required, opened a new log
I20260929 20:54:20.560673 11165 tablet_bootstrap.cc:492] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8: No bootstrap required, opened a new log
I20260929 20:54:20.560894 11165 ts_tablet_manager.cc:1397] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260929 20:54:20.561358 11166 ts_tablet_manager.cc:1397] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 20:54:20.567579 11165 raft_consensus.cc:348] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983547bdc948418f8f5db7ab4eecf5f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37091 } } peers { permanent_uuid: "125dc936467a444a91989ece5ffc17c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "0a9011c9b88e4a2884d491fc4a25a113" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40707 } }
I20260929 20:54:20.567916 11165 raft_consensus.cc:374] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:54:20.568004 11165 raft_consensus.cc:729] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125dc936467a444a91989ece5ffc17c8, State: Initialized, Role: FOLLOWER
I20260929 20:54:20.568379 11164 raft_consensus.cc:348] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983547bdc948418f8f5db7ab4eecf5f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37091 } } peers { permanent_uuid: "125dc936467a444a91989ece5ffc17c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "0a9011c9b88e4a2884d491fc4a25a113" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40707 } }
I20260929 20:54:20.568560 11165 consensus_queue.cc:260] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983547bdc948418f8f5db7ab4eecf5f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37091 } } peers { permanent_uuid: "125dc936467a444a91989ece5ffc17c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "0a9011c9b88e4a2884d491fc4a25a113" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40707 } }
I20260929 20:54:20.568759 11164 raft_consensus.cc:374] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:54:20.568851 11164 raft_consensus.cc:729] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a9011c9b88e4a2884d491fc4a25a113, State: Initialized, Role: FOLLOWER
I20260929 20:54:20.569562 11013 heartbeater.cc:503] Master 127.0.97.62:37939 was elected leader, sending a full tablet report...
I20260929 20:54:20.569486 11164 consensus_queue.cc:260] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983547bdc948418f8f5db7ab4eecf5f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37091 } } peers { permanent_uuid: "125dc936467a444a91989ece5ffc17c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "0a9011c9b88e4a2884d491fc4a25a113" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40707 } }
I20260929 20:54:20.570238 11165 ts_tablet_manager.cc:1428] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260929 20:54:20.571262 11166 raft_consensus.cc:348] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983547bdc948418f8f5db7ab4eecf5f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37091 } } peers { permanent_uuid: "125dc936467a444a91989ece5ffc17c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "0a9011c9b88e4a2884d491fc4a25a113" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40707 } }
I20260929 20:54:20.571658 11166 raft_consensus.cc:374] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:54:20.571748 11166 raft_consensus.cc:729] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983547bdc948418f8f5db7ab4eecf5f3, State: Initialized, Role: FOLLOWER
I20260929 20:54:20.572415 11166 consensus_queue.cc:260] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983547bdc948418f8f5db7ab4eecf5f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37091 } } peers { permanent_uuid: "125dc936467a444a91989ece5ffc17c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "0a9011c9b88e4a2884d491fc4a25a113" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40707 } }
I20260929 20:54:20.576149 11146 heartbeater.cc:503] Master 127.0.97.62:37939 was elected leader, sending a full tablet report...
I20260929 20:54:20.577440 11164 ts_tablet_manager.cc:1428] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
I20260929 20:54:20.578466 11166 ts_tablet_manager.cc:1428] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
W20260929 20:54:20.711011 11147 tablet.cc:2369] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:54:20.743662 11014 tablet.cc:2369] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:54:20.767812 10880 tablet.cc:2369] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:54:20.927579 11172 raft_consensus.cc:482] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:54:20.927888 11172 raft_consensus.cc:504] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983547bdc948418f8f5db7ab4eecf5f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37091 } } peers { permanent_uuid: "125dc936467a444a91989ece5ffc17c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "0a9011c9b88e4a2884d491fc4a25a113" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40707 } }
I20260929 20:54:20.929491 11172 leader_election.cc:288] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125dc936467a444a91989ece5ffc17c8 (127.0.97.2:33695), 0a9011c9b88e4a2884d491fc4a25a113 (127.0.97.1:40707)
I20260929 20:54:20.936225 11148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:20.929799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33695 (local address 127.0.97.3:38595)
0929 20:54:20.929985 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:20.930002 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:20.930111 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:20.930787 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:20.930793 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:20.930834 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:54:20.931110 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:20.931123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.932268 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.932272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:20.934085 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:20.934100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.935438 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.935444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:20.935686 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:20.935722 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:54:20.935806 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 20:54:20.935937 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":24}
I20260929 20:54:20.936336 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:20.930220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38595 (local address 127.0.97.2:33695)
0929 20:54:20.930415 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:20.930421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:20.930441 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:54:20.930516 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:20.930521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:20.930601 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:20.930709 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:54:20.931260 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.932119 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.934267 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.935276 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.935414 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:20.935505 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:20.935972 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 20:54:20.936098 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:54:20.937263 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:20.929882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50829 (local address 127.0.97.1:40707)
0929 20:54:20.930123 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:20.930129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:20.931136 (+ 1007us) server_negotiation.cc:407] Connection header received
0929 20:54:20.931284 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:20.931288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:20.931360 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:20.931452 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:54:20.931951 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.932695 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.935032 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.936241 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.936364 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:20.936456 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:20.936927 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 20:54:20.937045 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:54:20.937372 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:20.930422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40707 (local address 127.0.97.3:50829)
0929 20:54:20.931025 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0929 20:54:20.931044 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:54:20.931178 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:54:20.931514 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:54:20.931518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:54:20.931539 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:54:20.931803 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:20.931810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.932846 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.932852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:20.934872 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:54:20.934885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.936571 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.936578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:54:20.936731 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:20.936762 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:54:20.936887 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 20:54:20.937012 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":1,"thread_start_us":340,"threads_started":1}
I20260929 20:54:20.937263 10968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4dcb9d113a54a73a448018984a8fcb3" candidate_uuid: "983547bdc948418f8f5db7ab4eecf5f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125dc936467a444a91989ece5ffc17c8" is_pre_election: true
I20260929 20:54:20.937798 10968 raft_consensus.cc:2454] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 983547bdc948418f8f5db7ab4eecf5f3 in term 0.
I20260929 20:54:20.938179 10834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4dcb9d113a54a73a448018984a8fcb3" candidate_uuid: "983547bdc948418f8f5db7ab4eecf5f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a9011c9b88e4a2884d491fc4a25a113" is_pre_election: true
I20260929 20:54:20.938480 11037 leader_election.cc:302] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125dc936467a444a91989ece5ffc17c8, 983547bdc948418f8f5db7ab4eecf5f3; no voters:
I20260929 20:54:20.938726 10834 raft_consensus.cc:2454] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 983547bdc948418f8f5db7ab4eecf5f3 in term 0.
I20260929 20:54:20.938951 11172 raft_consensus.cc:2781] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:54:20.939129 11172 raft_consensus.cc:482] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:54:20.939241 11172 raft_consensus.cc:3037] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:54:20.942345 11172 raft_consensus.cc:504] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983547bdc948418f8f5db7ab4eecf5f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37091 } } peers { permanent_uuid: "125dc936467a444a91989ece5ffc17c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "0a9011c9b88e4a2884d491fc4a25a113" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40707 } }
I20260929 20:54:20.943066 11172 leader_election.cc:288] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [CANDIDATE]: Term 1 election: Requested vote from peers 125dc936467a444a91989ece5ffc17c8 (127.0.97.2:33695), 0a9011c9b88e4a2884d491fc4a25a113 (127.0.97.1:40707)
I20260929 20:54:20.943470 10834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4dcb9d113a54a73a448018984a8fcb3" candidate_uuid: "983547bdc948418f8f5db7ab4eecf5f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a9011c9b88e4a2884d491fc4a25a113"
I20260929 20:54:20.943466 10968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4dcb9d113a54a73a448018984a8fcb3" candidate_uuid: "983547bdc948418f8f5db7ab4eecf5f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125dc936467a444a91989ece5ffc17c8"
I20260929 20:54:20.943643 10834 raft_consensus.cc:3037] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:54:20.943661 10968 raft_consensus.cc:3037] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:54:20.945699 10834 raft_consensus.cc:2454] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 983547bdc948418f8f5db7ab4eecf5f3 in term 1.
I20260929 20:54:20.946058 11036 leader_election.cc:302] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a9011c9b88e4a2884d491fc4a25a113, 983547bdc948418f8f5db7ab4eecf5f3; no voters:
I20260929 20:54:20.946336 11172 raft_consensus.cc:2781] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:54:20.946666 10968 raft_consensus.cc:2454] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 983547bdc948418f8f5db7ab4eecf5f3 in term 1.
I20260929 20:54:20.946774 11172 raft_consensus.cc:686] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [term 1 LEADER]: Becoming Leader. State: Replica: 983547bdc948418f8f5db7ab4eecf5f3, State: Running, Role: LEADER
I20260929 20:54:20.947487 11172 consensus_queue.cc:237] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983547bdc948418f8f5db7ab4eecf5f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37091 } } peers { permanent_uuid: "125dc936467a444a91989ece5ffc17c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } } peers { permanent_uuid: "0a9011c9b88e4a2884d491fc4a25a113" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40707 } }
I20260929 20:54:20.951557 10690 catalog_manager.cc:5495] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 983547bdc948418f8f5db7ab4eecf5f3 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "983547bdc948418f8f5db7ab4eecf5f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983547bdc948418f8f5db7ab4eecf5f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125dc936467a444a91989ece5ffc17c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a9011c9b88e4a2884d491fc4a25a113" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40707 } health_report { overall_health: UNKNOWN } } }
I20260929 20:54:20.984604 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:20.979532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.0.97.3:37091)
0929 20:54:20.979739 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:20.979745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:20.979772 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:54:20.979960 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:20.979964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:20.980040 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:20.980173 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:54:20.980548 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.981252 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.982578 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:20.982962 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:20.983070 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:20.983539 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:54:20.984059 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:54:20.984197 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:20.984270 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 20:54:20.984352 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:54:20.992627 10834 raft_consensus.cc:1261] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113 [term 1 FOLLOWER]: Refusing update from remote peer 983547bdc948418f8f5db7ab4eecf5f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:54:20.992703 10968 raft_consensus.cc:1261] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8 [term 1 FOLLOWER]: Refusing update from remote peer 983547bdc948418f8f5db7ab4eecf5f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:54:20.993618 11174 consensus_queue.cc:1035] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a9011c9b88e4a2884d491fc4a25a113" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:54:20.993950 11172 consensus_queue.cc:1035] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125dc936467a444a91989ece5ffc17c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:54:21.009794 11177 mvcc.cc:205] Tried to move back new op lower bound from 7334769709016125440 to 7334769708844572672. Current Snapshot: MvccSnapshot[applied={T|T < 7334769709016125440}]
I20260929 20:54:21.023571 10690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:54:21.023854 10690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:54:21.027468 10690 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 198515da9d9444df99c356b45ac7e76b: Sending DeleteTablet for 3 replicas of tablet f4dcb9d113a54a73a448018984a8fcb3
I20260929 20:54:21.028761 10814 tablet_service.cc:1514] Processing DeleteTablet for tablet f4dcb9d113a54a73a448018984a8fcb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:54:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57600
I20260929 20:54:21.029079 11081 tablet_service.cc:1514] Processing DeleteTablet for tablet f4dcb9d113a54a73a448018984a8fcb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:54:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45626
I20260929 20:54:21.029079 10948 tablet_service.cc:1514] Processing DeleteTablet for tablet f4dcb9d113a54a73a448018984a8fcb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:54:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48420
I20260929 20:54:21.029860 11186 tablet_replica.cc:330] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113: stopping tablet replica
I20260929 20:54:21.030390 11186 raft_consensus.cc:2229] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:54:21.030555 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 10751
I20260929 20:54:21.031080 11186 raft_consensus.cc:2258] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:54:21.037952 11186 ts_tablet_manager.cc:1905] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:54:21.045210 11186 ts_tablet_manager.cc:1918] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:54:21.045403 11186 log.cc:1196] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/f4dcb9d113a54a73a448018984a8fcb3
I20260929 20:54:21.045928 11186 ts_tablet_manager.cc:1939] T f4dcb9d113a54a73a448018984a8fcb3 P 0a9011c9b88e4a2884d491fc4a25a113: Deleting consensus metadata
I20260929 20:54:21.047956 10676 catalog_manager.cc:4841] TS 0a9011c9b88e4a2884d491fc4a25a113 (127.0.97.1:40707): tablet f4dcb9d113a54a73a448018984a8fcb3 (table test-table [id=54b862c8cc854e6b9ebfae28dc9b1df7]) successfully deleted
Sep 29 20:54:21 dist-test-slave-m9g1 krb5kdc[10645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:54:21.048655 11187 tablet_replica.cc:330] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8: stopping tablet replica
I20260929 20:54:21.049137 11187 raft_consensus.cc:2229] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:54:21.049724 11187 raft_consensus.cc:2258] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:54:21.050549 11188 tablet_replica.cc:330] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3: stopping tablet replica
I20260929 20:54:21.051446 11188 raft_consensus.cc:2229] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:54:21.052090 11188 raft_consensus.cc:2258] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:54:21.052394 11187 ts_tablet_manager.cc:1905] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:54:21.054227 11188 ts_tablet_manager.cc:1905] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:54:21.057551 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:21.032130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.0.97.1:40707)
0929 20:54:21.032298 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:21.032303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:21.032825 (+ 522us) server_negotiation.cc:407] Connection header received
0929 20:54:21.033089 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:21.033095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:21.033172 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:21.033231 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:21.034917 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:21.036023 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:21.037782 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:21.038409 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:21.038508 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:21.049877 (+ 11369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:21.049896 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:21.049918 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:54:21.049991 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:21.053537 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:54:21.054470 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:54:21.054480 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:54:21.054484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:54:21.054646 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:54:21.055085 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:54:21.055089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:54:21.055093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:54:21.055508 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:21.055773 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:21.056974 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0929 20:54:21.057186 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:54:21.058250 10874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:54:21.059890 11187 ts_tablet_manager.cc:1918] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:54:21.060104 11187 log.cc:1196] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/f4dcb9d113a54a73a448018984a8fcb3
I20260929 20:54:21.060691 11187 ts_tablet_manager.cc:1939] T f4dcb9d113a54a73a448018984a8fcb3 P 125dc936467a444a91989ece5ffc17c8: Deleting consensus metadata
I20260929 20:54:21.061187 11188 ts_tablet_manager.cc:1918] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:54:21.061339 11188 log.cc:1196] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/f4dcb9d113a54a73a448018984a8fcb3
I20260929 20:54:21.061856 11188 ts_tablet_manager.cc:1939] T f4dcb9d113a54a73a448018984a8fcb3 P 983547bdc948418f8f5db7ab4eecf5f3: Deleting consensus metadata
I20260929 20:54:21.062860 10677 catalog_manager.cc:4841] TS 125dc936467a444a91989ece5ffc17c8 (127.0.97.2:33695): tablet f4dcb9d113a54a73a448018984a8fcb3 (table test-table [id=54b862c8cc854e6b9ebfae28dc9b1df7]) successfully deleted
I20260929 20:54:21.063424 10676 catalog_manager.cc:4841] TS 983547bdc948418f8f5db7ab4eecf5f3 (127.0.97.3:37091): tablet f4dcb9d113a54a73a448018984a8fcb3 (table test-table [id=54b862c8cc854e6b9ebfae28dc9b1df7]) successfully deleted
I20260929 20:54:21.175931 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 10751
I20260929 20:54:21.190937 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 10885
Sep 29 20:54:21 dist-test-slave-m9g1 krb5kdc[10645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:54:21.206485 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:21.191715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48430 (local address 127.0.97.2:33695)
0929 20:54:21.191889 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:21.191895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:21.191915 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:54:21.192072 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:21.192078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:21.192158 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:21.192272 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:21.193518 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:21.194678 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:21.195927 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:21.196563 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:21.196685 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:21.200431 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:21.200452 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:21.200474 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:54:21.200528 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:21.203952 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:54:21.204578 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:54:21.204588 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:54:21.204592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:54:21.204743 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:54:21.205021 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:54:21.205026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:54:21.205029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:54:21.205443 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:21.205731 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:21.205905 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 20:54:21.206112 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:54:21.207213 11008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:54:21.292783 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 10885
I20260929 20:54:21.306427 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 11018
Sep 29 20:54:21 dist-test-slave-m9g1 krb5kdc[10645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:54:21.321130 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:54:21.307145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45644 (local address 127.0.97.3:37091)
0929 20:54:21.307320 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:54:21.307326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:54:21.307404 (+ 78us) server_negotiation.cc:407] Connection header received
0929 20:54:21.307575 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:54:21.307584 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:54:21.307665 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:54:21.307758 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:54:21.308901 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:21.309729 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:21.311087 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:54:21.311556 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:54:21.311663 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:54:21.315620 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:54:21.315632 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:54:21.315646 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:54:21.315683 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:54:21.318579 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:54:21.319164 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:54:21.319171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:54:21.319173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:54:21.319323 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:54:21.319662 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:54:21.319665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:54:21.319667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:54:21.319995 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:54:21.320288 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 20:54:21.320547 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 20:54:21.320744 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:54:21.321776 11141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:54:21.439910 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 11018
I20260929 20:54:21.457079 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 10658
I20260929 20:54:21.458317 10720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:54:21.562008 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 10658
2026-09-29T20:54:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4265 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:54:21 dist-test-slave-m9g1 krb5kdc[11196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:54:21 dist-test-slave-m9g1 krb5kdc[11196](info): set up 2 sockets
Sep 29 20:54:21 dist-test-slave-m9g1 krb5kdc[11196](info): commencing operation
krb5kdc: starting...
W20260929 20:54:22.566817 388 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:54:22 dist-test-slave-m9g1 krb5kdc[11196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:54:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:54:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260929 20:54:22.854765 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPL7Duu/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/postgres -l logfile start
2026-09-29 20:54:25.105 UTC [11226] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 20:54:25.105 UTC [11226] LOG: listening on IPv4 address "127.0.97.20", port 48607
2026-09-29 20:54:25.108 UTC [11226] LOG: listening on Unix socket "/tmp/.s.PGSQL.48607"
2026-09-29 20:54:25.129 UTC [11229] LOG: database system was shut down at 2026-09-29 20:54:23 UTC
2026-09-29 20:54:25.132 UTC [11226] LOG: database system is ready to accept connections
I20260929 20:54:26.018446 388 mini_postgres.cc:96] Postgres bound to 48607
2026-09-29 20:54:26.034 UTC [11237] FATAL: database "slave" does not exist
127.0.97.20:48607 - accepting connections
I20260929 20:54:26.035287 388 mini_ranger.cc:161] Starting Ranger...
I20260929 20:54:26.060758 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 20:54:26.505441 388 mini_ranger.cc:87] Created ranger Postgres database
I20260929 20:54:26.505594 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ranger-admin
2026-09-29 20:54:26,799 [I] DB FLAVOR :POSTGRES
2026-09-29 20:54:26,800 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:54:26,800 [I] Checking connection..
2026-09-29 20:54:26,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 20:54:27,036 [I] Checking connection passed.
2026-09-29 20:54:27,036 [I] --------- Verifying version history table ---------
2026-09-29 20:54:27,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:54:27,270 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:54:27,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:54:27,501 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:54:27,502 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 20:54:27,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 20:54:27.732 UTC [11301] WARNING: there is no transaction in progress
2026-09-29 20:54:27,738 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 20:54:27,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:54:27,988 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 20:54:27,989 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:54:27,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 20:54:28,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 20:54:28,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-m9g1.c.gcp-upstream.internal','N') ;"
2026-09-29 20:54:28,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:54:28,943 [I] Table x_portal_user does not exist in database ranger
2026-09-29 20:54:28,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:54:29,187 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 20:54:29,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:54:29,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:54:29,686 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 20:54:29,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 20:54:30.587 UTC [11409] WARNING: there is no transaction in progress
2026-09-29 20:54:30.602 UTC [11409] WARNING: there is no transaction in progress
2026-09-29 20:54:30.617 UTC [11409] WARNING: there is no transaction in progress
2026-09-29 20:54:30.632 UTC [11409] WARNING: there is no transaction in progress
2026-09-29 20:54:30.646 UTC [11409] WARNING: there is no transaction in progress
2026-09-29 20:54:30.662 UTC [11409] WARNING: there is no transaction in progress
2026-09-29 20:54:30.713 UTC [11409] WARNING: there is no transaction in progress
2026-09-29 20:54:30.719 UTC [11409] WARNING: there is no transaction in progress
2026-09-29 20:54:30.733 UTC [11409] WARNING: there is no transaction in progress
2026-09-29 20:54:30.742 UTC [11409] WARNING: there is no transaction in progress
2026-09-29 20:54:30.752 UTC [11409] WARNING: there is no transaction in progress
2026-09-29 20:54:30.766 UTC [11409] WARNING: there is no transaction in progress
2026-09-29 20:54:30.775 UTC [11409] WARNING: there is no transaction in progress
2026-09-29 20:54:30.781 UTC [11409] WARNING: there is no transaction in progress
2026-09-29 20:54:30.787 UTC [11409] WARNING: there is no transaction in progress
2026-09-29 20:54:30.793 UTC [11409] WARNING: there is no transaction in progress
2026-09-29 20:54:31,375 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 20:54:31,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 20:54:31,617 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 20:54:31,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:54:31,852 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 20:54:31,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:54:32,106 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 20:54:32,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:54:32,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:54:32,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-m9g1.c.gcp-upstream.internal';"
2026-09-29 20:54:32,824 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 20:54:32,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 20:54:33,068 [I] DB_PATCHES have already been applied
I20260929 20:54:33.074688 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPL7Duu/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 20:54:33.074781 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260929 20:54:33.089545 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:59763
Sep 29, 2026 8:54:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:54:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:54:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 8:54:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 8:54:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 8:54:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 8:54:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 20:54:34.133068 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.098s user 0.008s sys 0.032s
Sep 29, 2026 8:54:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 8:54:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 8:54:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 8:54:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 8:54:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 8:54:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 8:54:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 8:54:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 8:54:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 8:54:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 8:54:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 8:54:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 8:54:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 8:54:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 8:54:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 8:54:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 8:54:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 8:54:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 20:54:55.759331 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 20:54:55 dist-test-slave-m9g1 krb5kdc[11196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715295, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 20:54:55.928287 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 20:54:55.950064 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 20:54:56.790136 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 20:54:56.790333 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ranger-kms
I20260929 20:54:56.790462 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:48607
2026-09-29 20:54:57,172 [I] DB FLAVOR :POSTGRES
2026-09-29 20:54:57,172 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:54:57,172 [I] Checking connection
2026-09-29 20:54:57,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 20:54:57,399 [I] connection success
2026-09-29 20:54:57,399 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 20:54:57,399 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 20:54:57,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 20:54:57,627 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 20:54:57,627 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:54:57,627 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 20:54:57,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48607/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 20:54:57,873 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 20:54:58.339721 388 mini_ranger_kms.cc:310] Created kms service
I20260929 20:54:58.592469 388 mini_ranger_kms.cc:326] Created kudu user
I20260929 20:54:58.634371 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 20:54:58 dist-test-slave-m9g1 krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715295, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:54:59.010749 388 mini_ranger_kms.cc:384] Added ranger policy
I20260929 20:54:59.011029 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 20:54:59.011089 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260929 20:54:59.032333 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11589
I20260929 20:54:59.032536 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:51971
Sep 29, 2026 8:55:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:55:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:55:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 8:55:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 8:55:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 8:55:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 20:55:01.100226 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.172s user 0.098s sys 0.305s
I20260929 20:55:01.100471 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 20:55:01.100653 388 mini_ranger_kms.cc:401] 127.0.97.20:51971/kms/v1/keys
Sep 29 20:55:01 dist-test-slave-m9g1 krb5kdc[11196](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790715301, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 20:55:01 dist-test-slave-m9g1 krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715301, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 20:55:02 dist-test-slave-m9g1 krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715301, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 20:55:04 dist-test-slave-m9g1 krb5kdc[11196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:55:04.293597 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45219
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45219
--ranger_config_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51971/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:55:04.631896 11640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:55:04.632212 11640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:55:04.632339 11640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:55:04.639854 11640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:55:04.639987 11640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:55:04.640061 11640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:55:04.640136 11640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:55:04.640197 11640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:55:04.651054 11640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:51971/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45219
--ranger_config_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:55:04.652396 11640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:55:04.653682 11640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:55:04.665441 11645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:04.665562 11646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:04.665490 11648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:55:04.666697 11640 server_base.cc:1034] running on GCE node
I20260929 20:55:04.667483 11640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:55:04.668908 11640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:55:04.670117 11640 hybrid_clock.cc:648] HybridClock initialized: now 1790715304670070 us; error 41 us; skew 500 ppm
I20260929 20:55:04.670480 11640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:55:04 dist-test-slave-m9g1 krb5kdc[11196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:55:04.677997 11640 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:55:04.679591 11640 webserver.cc:460] Webserver started at http://127.0.97.62:34533/ using document root <none> and password file <none>
I20260929 20:55:04.680289 11640 fs_manager.cc:359] Metadata directory not provided
I20260929 20:55:04.680406 11640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:55:04.680713 11640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:55:04 dist-test-slave-m9g1 krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:55:04.745164 11640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "93d96cc371d14e8a8810f33f2dc00681"
format_stamp: "Formatted at 2026-09-29 20:55:04 on dist-test-slave-m9g1"
server_key: "12a1e79c61bb21e3687ca1e80b201e8d"
server_key_iv: "8c49ef2bdc2832eb790105a2d927cd45"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:04.746030 11640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "93d96cc371d14e8a8810f33f2dc00681"
format_stamp: "Formatted at 2026-09-29 20:55:04 on dist-test-slave-m9g1"
server_key: "12a1e79c61bb21e3687ca1e80b201e8d"
server_key_iv: "8c49ef2bdc2832eb790105a2d927cd45"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:04.751364 11640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:55:04.767635 11655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:04.769232 11640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:55:04.769425 11640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "93d96cc371d14e8a8810f33f2dc00681"
format_stamp: "Formatted at 2026-09-29 20:55:04 on dist-test-slave-m9g1"
server_key: "12a1e79c61bb21e3687ca1e80b201e8d"
server_key_iv: "8c49ef2bdc2832eb790105a2d927cd45"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:04.769663 11640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:55:04.792524 11640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:55:04.796424 11640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:55:04.796689 11640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:55:04.815558 11640 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45219
I20260929 20:55:04.815573 11706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45219 every 8 connection(s)
I20260929 20:55:04.816963 11640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:55:04.821317 11707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:55:04.822681 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 11640
I20260929 20:55:04.822876 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:55:04.823657 388 external_mini_cluster.cc:1428] Setting key 388bcdb64b910bc942568bc2210a34a7
I20260929 20:55:04.836246 11707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681: Bootstrap starting.
I20260929 20:55:04.840554 11707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681: Neither blocks nor log segments found. Creating new log.
I20260929 20:55:04.842403 11707 log.cc:824] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681: Log is configured to *not* fsync() on all Append() calls
I20260929 20:55:04.846886 11707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681: No bootstrap required, opened a new log
Sep 29 20:55:04 dist-test-slave-m9g1 krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:55:04.854840 11707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d96cc371d14e8a8810f33f2dc00681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45219 } }
I20260929 20:55:04.855489 11707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:55:04.855584 11707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93d96cc371d14e8a8810f33f2dc00681, State: Initialized, Role: FOLLOWER
I20260929 20:55:04.856161 11707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d96cc371d14e8a8810f33f2dc00681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45219 } }
I20260929 20:55:04.856380 11707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:55:04.856510 11707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:55:04.856685 11707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:55:04.859341 11707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d96cc371d14e8a8810f33f2dc00681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45219 } }
I20260929 20:55:04.859764 11710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:04.825872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40562 (local address 127.0.97.62:45219)
0929 20:55:04.838752 (+ 12880us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:04.838769 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:04.839181 (+ 412us) server_negotiation.cc:407] Connection header received
0929 20:55:04.841673 (+ 2492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:04.841721 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:04.842339 (+ 618us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:04.842780 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:55:04.844311 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:04.846056 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:04.847395 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:04.848371 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:04.848552 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:04.852264 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:55:04.852275 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:55:04.852300 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:55:04.852345 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:55:04.855667 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:04.856363 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:04.856376 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:04.856383 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:04.856562 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:04.856963 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:04.856966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:04.856968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:04.857523 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:55:04.857745 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:04.858232 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 20:55:04.858589 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12613,"thread_start_us":12384,"threads_started":1}
I20260929 20:55:04.859870 11707 leader_election.cc:302] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93d96cc371d14e8a8810f33f2dc00681; no voters:
I20260929 20:55:04.860535 11707 leader_election.cc:288] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:55:04.860651 11712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:55:04.863101 11712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681 [term 1 LEADER]: Becoming Leader. State: Replica: 93d96cc371d14e8a8810f33f2dc00681, State: Running, Role: LEADER
I20260929 20:55:04.863711 11712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d96cc371d14e8a8810f33f2dc00681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45219 } }
I20260929 20:55:04.864058 11707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:55:04.867307 11713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93d96cc371d14e8a8810f33f2dc00681" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d96cc371d14e8a8810f33f2dc00681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45219 } } }
I20260929 20:55:04.867292 11714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93d96cc371d14e8a8810f33f2dc00681. Latest consensus state: current_term: 1 leader_uuid: "93d96cc371d14e8a8810f33f2dc00681" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d96cc371d14e8a8810f33f2dc00681" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45219 } } }
I20260929 20:55:04.867803 11713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681 [sys.catalog]: This master's current role is: LEADER
I20260929 20:55:04.867815 11714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681 [sys.catalog]: This master's current role is: LEADER
I20260929 20:55:04.868629 11719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:55:04.879914 11719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:55:04.880673 11707 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 20:55:04.894942 11719 catalog_manager.cc:1347] Generated new cluster ID: d22b8b80cfd24d5aa80a39a75d470d84
I20260929 20:55:04.895085 11719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:55:04.924800 11719 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:55:04.926838 11719 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:55:04.939067 11719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681: Generated new TSK 0
I20260929 20:55:04.940191 11719 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:55:05.627018 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51971/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45219
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:55:05 dist-test-slave-m9g1 krb5kdc[11196](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790715305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 20:55:06.076207 11752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:55:06.076640 11752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:55:06.076814 11752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:55:06.088393 11752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:55:06.088552 11752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:55:06.088745 11752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:55:06.104130 11752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:51971/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:55:06.105953 11752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:55:06.107720 11752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:55:06.129815 11762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:06.130347 11763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:06.143460 11765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:55:06.143764 11752 server_base.cc:1034] running on GCE node
I20260929 20:55:06.144640 11752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:55:06.145996 11752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:55:06.147293 11752 hybrid_clock.cc:648] HybridClock initialized: now 1790715306147234 us; error 72 us; skew 500 ppm
I20260929 20:55:06.147707 11752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:55:06 dist-test-slave-m9g1 krb5kdc[11196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:55:06.152038 11752 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:55:06.154484 11752 webserver.cc:460] Webserver started at http://127.0.97.1:37005/ using document root <none> and password file <none>
I20260929 20:55:06.155438 11752 fs_manager.cc:359] Metadata directory not provided
I20260929 20:55:06.155576 11752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:55:06.155915 11752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:55:06 dist-test-slave-m9g1 krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:55:06.185495 11752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "0b477eaea0824b39a4d919a695850b4e"
format_stamp: "Formatted at 2026-09-29 20:55:06 on dist-test-slave-m9g1"
server_key: "50875682263cd991d340d72ae7cc8c20"
server_key_iv: "aeb3ffcd9de412ac86d2fdeb604ed2f9"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:06.186805 11752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0b477eaea0824b39a4d919a695850b4e"
format_stamp: "Formatted at 2026-09-29 20:55:06 on dist-test-slave-m9g1"
server_key: "50875682263cd991d340d72ae7cc8c20"
server_key_iv: "aeb3ffcd9de412ac86d2fdeb604ed2f9"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:06.194226 11752 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260929 20:55:06.218310 11772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:06.224018 11752 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260929 20:55:06.224349 11752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "0b477eaea0824b39a4d919a695850b4e"
format_stamp: "Formatted at 2026-09-29 20:55:06 on dist-test-slave-m9g1"
server_key: "50875682263cd991d340d72ae7cc8c20"
server_key_iv: "aeb3ffcd9de412ac86d2fdeb604ed2f9"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:06.224705 11752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:55:06.247896 11752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:55:06.253744 11752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:55:06.254138 11752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:55:06.255437 11752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:55:06.257397 11752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:55:06.257483 11752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:06.257647 11752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:55:06.257707 11752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:55:06 dist-test-slave-m9g1 krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:55:06.764990 11752 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34577
I20260929 20:55:06.764990 11884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34577 every 8 connection(s)
I20260929 20:55:06.766408 11752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:55:06.774037 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 11752
I20260929 20:55:06.774277 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:55:06.775089 388 external_mini_cluster.cc:1428] Setting key 7aad7ca80c16f3bbf96afd00cde6a60a
Sep 29 20:55:06 dist-test-slave-m9g1 krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:55:06.830602 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:06.770713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58977 (local address 127.0.97.62:45219)
0929 20:55:06.805506 (+ 34793us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:06.805515 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:06.810239 (+ 4724us) server_negotiation.cc:407] Connection header received
0929 20:55:06.811788 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:06.811794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:06.811894 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:06.811992 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:55:06.814495 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:06.815591 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:06.818505 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:06.819168 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:06.819252 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:06.823607 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:55:06.823614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:55:06.823619 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:55:06.823670 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:55:06.826968 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:06.827838 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:06.827843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:06.827847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:06.827976 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:06.828422 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:06.828426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:06.828429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:06.828750 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:55:06.828871 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:06.829923 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0929 20:55:06.830158 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34628,"thread_start_us":34812,"threads_started":1}
I20260929 20:55:06.833611 11887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:06.769962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45219 (local address 127.0.97.1:58977)
0929 20:55:06.807059 (+ 37097us) negotiation.cc:107] Waiting for socket to connect
0929 20:55:06.807136 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 20:55:06.809994 (+ 2858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:55:06.812741 (+ 2747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:55:06.812754 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:55:06.813413 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:55:06.814293 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:06.814320 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:06.815802 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:06.815807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:06.818358 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:06.818373 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:06.819583 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:06.819590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:06.819854 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:06.820629 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:55:06.820658 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:55:06.823337 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:55:06.827126 (+ 3789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:55:06.827135 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:55:06.827155 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:55:06.827713 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:55:06.828095 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:55:06.828100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:55:06.828103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:55:06.828315 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:55:06.828960 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:55:06.828969 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:55:06.829459 (+ 490us) client_negotiation.cc:769] Sending connection context
0929 20:55:06.831195 (+ 1736us) client_negotiation.cc:240] Negotiation successful
0929 20:55:06.831752 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36591,"thread_start_us":40183,"threads_started":1}
I20260929 20:55:06.835862 11885 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45219
I20260929 20:55:06.836233 11885 heartbeater.cc:464] Registering TS with master...
I20260929 20:55:06.837159 11885 heartbeater.cc:511] Master 127.0.97.62:45219 requested a full tablet report, sending...
I20260929 20:55:06.839856 11672 ts_manager.cc:194] Registered new tserver with Master: 0b477eaea0824b39a4d919a695850b4e (127.0.97.1:34577)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:55:06.841850 11672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Sep 29 20:55:06 dist-test-slave-m9g1 krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
W20260929 20:55:06.878949 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.2: real 0.101s user 0.002s sys 0.015s
I20260929 20:55:06.879887 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51971/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45219
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:55:07.306262 11893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:55:07.306523 11893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:55:07.306648 11893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:55:07.314378 11893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:55:07.314495 11893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:55:07.314638 11893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:55:07.325172 11893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:51971/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:55:07.326491 11893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:55:07.327747 11893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:55:07.343302 11905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:07.343437 11906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:07.343540 11908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:55:07.343966 11893 server_base.cc:1034] running on GCE node
I20260929 20:55:07.344614 11893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:55:07.345646 11893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:55:07.346855 11893 hybrid_clock.cc:648] HybridClock initialized: now 1790715307346803 us; error 48 us; skew 500 ppm
I20260929 20:55:07.347177 11893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:55:07 dist-test-slave-m9g1 krb5kdc[11196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:55:07.350369 11893 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:55:07.351845 11893 webserver.cc:460] Webserver started at http://127.0.97.2:44789/ using document root <none> and password file <none>
I20260929 20:55:07.352515 11893 fs_manager.cc:359] Metadata directory not provided
I20260929 20:55:07.352627 11893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:55:07.352910 11893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:55:07 dist-test-slave-m9g1 krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:55:07.369899 11893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "0f689c7085f84ed29351ba6049816ae4"
format_stamp: "Formatted at 2026-09-29 20:55:07 on dist-test-slave-m9g1"
server_key: "8d36d3dceb2cd7dfde674969f340c339"
server_key_iv: "26a64624898ae1c2d96a995757fcb3d6"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:07.371358 11893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0f689c7085f84ed29351ba6049816ae4"
format_stamp: "Formatted at 2026-09-29 20:55:07 on dist-test-slave-m9g1"
server_key: "8d36d3dceb2cd7dfde674969f340c339"
server_key_iv: "26a64624898ae1c2d96a995757fcb3d6"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:07.378448 11893 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260929 20:55:07.396039 11915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:07.397622 11893 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:55:07.397797 11893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "0f689c7085f84ed29351ba6049816ae4"
format_stamp: "Formatted at 2026-09-29 20:55:07 on dist-test-slave-m9g1"
server_key: "8d36d3dceb2cd7dfde674969f340c339"
server_key_iv: "26a64624898ae1c2d96a995757fcb3d6"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:07.398008 11893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:55:07.420635 11893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:55:07.424590 11893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:55:07.424858 11893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:55:07.425817 11893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:55:07.427209 11893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:55:07.427278 11893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:07.427389 11893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:55:07.427438 11893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:07.465054 11893 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44623
I20260929 20:55:07.465144 12027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44623 every 8 connection(s)
I20260929 20:55:07.466361 11893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:55:07.468726 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 11893
I20260929 20:55:07.468883 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:55:07.469374 388 external_mini_cluster.cc:1428] Setting key a71cf9f6c106fdf5f44d6343d96ae913
Sep 29 20:55:07 dist-test-slave-m9g1 krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:55:07.490450 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:07.468861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50225 (local address 127.0.97.62:45219)
0929 20:55:07.469289 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:07.469295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:07.470537 (+ 1242us) server_negotiation.cc:407] Connection header received
0929 20:55:07.473521 (+ 2984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:07.473526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:07.473607 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:07.473717 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:55:07.475647 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:07.476644 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:07.478988 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:07.479567 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:07.479680 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:07.483809 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:55:07.483816 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:55:07.483820 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:55:07.483856 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:55:07.487324 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:07.488062 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:07.488068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:07.488071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:07.488221 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:07.488640 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:07.488645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:07.488648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:07.489009 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:55:07.489144 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:07.489813 (+ 669us) server_negotiation.cc:299] Negotiation successful
0929 20:55:07.490045 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":216,"threads_started":1}
I20260929 20:55:07.491199 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:07.469496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45219 (local address 127.0.97.2:50225)
0929 20:55:07.470361 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 20:55:07.470417 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 20:55:07.473210 (+ 2793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:55:07.473993 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:55:07.474005 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:55:07.474636 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:55:07.475448 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:07.475475 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:07.476806 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:07.476811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:07.478827 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:07.478842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:07.479792 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:07.479798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:07.480086 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:07.480817 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:55:07.480845 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:55:07.483551 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:55:07.487474 (+ 3923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:55:07.487483 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:55:07.487503 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:55:07.487939 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:55:07.488314 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:55:07.488317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:55:07.488319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:55:07.488510 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:55:07.489122 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:55:07.489128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:55:07.489438 (+ 310us) client_negotiation.cc:769] Sending connection context
0929 20:55:07.489666 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 20:55:07.490037 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":356,"threads_started":1}
I20260929 20:55:07.492599 12028 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 20:55:07.492918 12028 heartbeater.cc:464] Registering TS with master...
I20260929 20:55:07.493645 12028 heartbeater.cc:511] Master 127.0.97.62:45219 requested a full tablet report, sending...
I20260929 20:55:07.495093 11672 ts_manager.cc:194] Registered new tserver with Master: 0f689c7085f84ed29351ba6049816ae4 (127.0.97.2:44623)
I20260929 20:55:07.496628 11672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:55:07.517305 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51971/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45219
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:55:07.837388 12034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:55:07.837651 12034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:55:07.837739 12034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:55:07.844750 12034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:55:07.844846 12034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:55:07.844987 12034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:55:07.845674 11885 heartbeater.cc:503] Master 127.0.97.62:45219 was elected leader, sending a full tablet report...
I20260929 20:55:07.854847 12034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:51971/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:55:07.856061 12034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:55:07.857275 12034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:55:07.869803 12040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:07.869823 12042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:55:07.870343 12034 server_base.cc:1034] running on GCE node
W20260929 20:55:07.870622 12039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:55:07.871253 12034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:55:07.872308 12034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:55:07.873508 12034 hybrid_clock.cc:648] HybridClock initialized: now 1790715307873464 us; error 39 us; skew 500 ppm
I20260929 20:55:07.873819 12034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:55:07 dist-test-slave-m9g1 krb5kdc[11196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:55:07.876885 12034 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:55:07.878314 12034 webserver.cc:460] Webserver started at http://127.0.97.3:34583/ using document root <none> and password file <none>
I20260929 20:55:07.879014 12034 fs_manager.cc:359] Metadata directory not provided
I20260929 20:55:07.879112 12034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:55:07.879361 12034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:55:07 dist-test-slave-m9g1 krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:55:07.896171 12034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "2df36ca7a2634d1bbf049323c0061255"
format_stamp: "Formatted at 2026-09-29 20:55:07 on dist-test-slave-m9g1"
server_key: "df3213e2c3cc6acae28a470b368024ce"
server_key_iv: "7fc3d9c69bcef325e13ed70c80052d57"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:07.897228 12034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2df36ca7a2634d1bbf049323c0061255"
format_stamp: "Formatted at 2026-09-29 20:55:07 on dist-test-slave-m9g1"
server_key: "df3213e2c3cc6acae28a470b368024ce"
server_key_iv: "7fc3d9c69bcef325e13ed70c80052d57"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:07.904036 12034 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260929 20:55:07.918646 12049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:07.920131 12034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:55:07.920288 12034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "2df36ca7a2634d1bbf049323c0061255"
format_stamp: "Formatted at 2026-09-29 20:55:07 on dist-test-slave-m9g1"
server_key: "df3213e2c3cc6acae28a470b368024ce"
server_key_iv: "7fc3d9c69bcef325e13ed70c80052d57"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:07.920511 12034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:55:07.937601 12034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:55:07.941346 12034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:55:07.941606 12034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:55:07.942525 12034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:55:07.944216 12034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:55:07.944283 12034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:07.944386 12034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:55:07.944432 12034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:07.975752 12034 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38993
I20260929 20:55:07.975773 12161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38993 every 8 connection(s)
I20260929 20:55:07.977030 12034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:55:07.977464 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 12034
I20260929 20:55:07.977615 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:55:07.978221 388 external_mini_cluster.cc:1428] Setting key f51839c8e9e640e0c8a06d211caa0ee4
Sep 29 20:55:07 dist-test-slave-m9g1 krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:55:07.996588 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:07.979700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56125 (local address 127.0.97.62:45219)
0929 20:55:07.979893 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:07.979899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:07.981293 (+ 1394us) server_negotiation.cc:407] Connection header received
0929 20:55:07.983155 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:07.983160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:07.983253 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:07.983340 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:55:07.985163 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:07.985854 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:07.987406 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:07.987834 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:07.987929 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:07.991157 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:55:07.991162 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:55:07.991165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:55:07.991202 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:55:07.994008 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:07.994671 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:07.994675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:07.994677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:07.994797 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:07.995171 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:07.995174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:07.995176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:07.995426 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:55:07.995532 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:07.996147 (+ 615us) server_negotiation.cc:299] Negotiation successful
0929 20:55:07.996323 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:55:07.997469 12164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:07.980307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45219 (local address 127.0.97.3:56125)
0929 20:55:07.981126 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0929 20:55:07.981185 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 20:55:07.982874 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:55:07.983533 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:55:07.983541 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:55:07.983990 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:55:07.985006 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:07.985026 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:07.985998 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:07.986001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:07.987268 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:07.987279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:07.988079 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:07.988085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:07.988288 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:07.988872 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:55:07.988893 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:55:07.990917 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:55:07.994119 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:55:07.994126 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:55:07.994140 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:55:07.994578 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:55:07.994872 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:55:07.994875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:55:07.994877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:55:07.995088 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:55:07.995512 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:55:07.995517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:55:07.995806 (+ 289us) client_negotiation.cc:769] Sending connection context
0929 20:55:07.996046 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 20:55:07.996402 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":350,"threads_started":1}
I20260929 20:55:07.998751 12162 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45219
I20260929 20:55:07.999029 12162 heartbeater.cc:464] Registering TS with master...
I20260929 20:55:07.999660 12162 heartbeater.cc:511] Master 127.0.97.62:45219 requested a full tablet report, sending...
I20260929 20:55:08.000931 11672 ts_manager.cc:194] Registered new tserver with Master: 2df36ca7a2634d1bbf049323c0061255 (127.0.97.3:38993)
I20260929 20:55:08.001703 11672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56125
I20260929 20:55:08.005544 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:55:08.021845 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:08.009643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40576 (local address 127.0.97.62:45219)
0929 20:55:08.009791 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:08.009796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:08.010263 (+ 467us) server_negotiation.cc:407] Connection header received
0929 20:55:08.010429 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:08.010434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:08.010502 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:08.010582 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:55:08.011888 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.013032 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.014331 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.014731 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.014836 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:08.017021 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:55:08.017027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:55:08.017030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:55:08.017061 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:55:08.019452 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:08.020063 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:08.020069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:08.020073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:08.020210 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:08.020595 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:08.020598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:08.020599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:08.020841 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:55:08.021038 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:08.021369 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 20:55:08.021540 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:55:08.025841 11672 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:55:08.032861 11672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:55:08.067267 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:08.045695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46454 (local address 127.0.97.1:34577)
0929 20:55:08.056508 (+ 10813us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:08.056519 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:08.057839 (+ 1320us) server_negotiation.cc:407] Connection header received
0929 20:55:08.057979 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:08.057988 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:08.058283 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:08.058468 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:55:08.059408 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.061813 (+ 2405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.064812 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.066060 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.066176 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:08.066440 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:08.066735 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 20:55:08.066905 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10614,"thread_start_us":10451,"threads_started":1}
I20260929 20:55:08.067649 12176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:08.045272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34577 (local address 127.0.0.1:46454)
0929 20:55:08.057124 (+ 11852us) negotiation.cc:107] Waiting for socket to connect
0929 20:55:08.057171 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:55:08.057371 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:55:08.058755 (+ 1384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:55:08.058763 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:55:08.058794 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:55:08.059206 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:08.059219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.062003 (+ 2784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.062010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:08.064638 (+ 2628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:08.064656 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.066273 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.066279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:08.066482 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:08.066543 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 20:55:08.066669 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 20:55:08.067229 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11562,"mutex_wait_us":1,"thread_start_us":17524,"threads_started":1}
I20260929 20:55:08.068136 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:08.045403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38993 (local address 127.0.0.1:57544)
0929 20:55:08.057911 (+ 12508us) negotiation.cc:107] Waiting for socket to connect
0929 20:55:08.057931 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:55:08.058068 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:55:08.059372 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:55:08.059378 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:55:08.059405 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:55:08.060349 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:08.060363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.063333 (+ 2970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.063337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:08.065496 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:08.065510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.067364 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.067374 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:08.067567 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:08.067605 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 20:55:08.067671 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 20:55:08.067793 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12377,"thread_start_us":12210,"threads_started":1}
I20260929 20:55:08.069193 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:08.046401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57544 (local address 127.0.97.3:38993)
0929 20:55:08.058752 (+ 12351us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:08.058761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:08.058796 (+ 35us) server_negotiation.cc:407] Connection header received
0929 20:55:08.058892 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:08.058899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:08.059149 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:08.060558 (+ 1409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:55:08.060855 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.062543 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.065708 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.067131 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.067942 (+ 811us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:08.068151 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:08.068219 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 20:55:08.068370 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12160,"thread_start_us":11986,"threads_started":1}
I20260929 20:55:08.071208 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:08.046849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57606 (local address 127.0.97.2:44623)
0929 20:55:08.063364 (+ 16515us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:08.063374 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:08.063400 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:55:08.063505 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:08.063512 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:08.063769 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:08.063954 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:55:08.064788 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.066287 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.068829 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.070103 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.070273 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:08.070496 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:08.070768 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 20:55:08.070940 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16357,"thread_start_us":16262,"threads_started":1}
I20260929 20:55:08.071825 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:08.045746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44623 (local address 127.0.0.1:57606)
0929 20:55:08.063061 (+ 17315us) negotiation.cc:107] Waiting for socket to connect
0929 20:55:08.063080 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:55:08.063185 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:55:08.064167 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:55:08.064172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:55:08.064199 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:55:08.064592 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:08.064603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.066443 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.066448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:08.068687 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:08.068698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.070335 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.070342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:08.070505 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:08.070537 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:55:08.070626 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 20:55:08.071505 (+ 879us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17190,"thread_start_us":10948,"threads_started":1}
I20260929 20:55:08.072245 11820 tablet_service.cc:1467] Processing CreateTablet for tablet 761dfa13616241919fec671c6381a501 (DEFAULT_TABLE table=test-table [id=26fb42d1fafc41358a47ff3490ad677a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:55:08.072587 12097 tablet_service.cc:1467] Processing CreateTablet for tablet 761dfa13616241919fec671c6381a501 (DEFAULT_TABLE table=test-table [id=26fb42d1fafc41358a47ff3490ad677a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:55:08.073726 11820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761dfa13616241919fec671c6381a501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:55:08.073905 12097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761dfa13616241919fec671c6381a501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:55:08.075349 11963 tablet_service.cc:1467] Processing CreateTablet for tablet 761dfa13616241919fec671c6381a501 (DEFAULT_TABLE table=test-table [id=26fb42d1fafc41358a47ff3490ad677a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:55:08.076637 11963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761dfa13616241919fec671c6381a501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:55:08.084519 12180 tablet_bootstrap.cc:492] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e: Bootstrap starting.
I20260929 20:55:08.088559 12181 tablet_bootstrap.cc:492] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255: Bootstrap starting.
I20260929 20:55:08.088608 12180 tablet_bootstrap.cc:654] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e: Neither blocks nor log segments found. Creating new log.
I20260929 20:55:08.089818 12180 log.cc:824] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:55:08.091711 12182 tablet_bootstrap.cc:492] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4: Bootstrap starting.
I20260929 20:55:08.092226 12181 tablet_bootstrap.cc:654] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255: Neither blocks nor log segments found. Creating new log.
I20260929 20:55:08.093397 12181 log.cc:824] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255: Log is configured to *not* fsync() on all Append() calls
I20260929 20:55:08.094035 12180 tablet_bootstrap.cc:492] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e: No bootstrap required, opened a new log
I20260929 20:55:08.094309 12180 ts_tablet_manager.cc:1397] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 20:55:08.095664 12182 tablet_bootstrap.cc:654] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4: Neither blocks nor log segments found. Creating new log.
I20260929 20:55:08.096797 12182 log.cc:824] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4: Log is configured to *not* fsync() on all Append() calls
I20260929 20:55:08.096894 12181 tablet_bootstrap.cc:492] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255: No bootstrap required, opened a new log
I20260929 20:55:08.097122 12181 ts_tablet_manager.cc:1397] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 20:55:08.099963 12182 tablet_bootstrap.cc:492] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4: No bootstrap required, opened a new log
I20260929 20:55:08.100226 12182 ts_tablet_manager.cc:1397] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260929 20:55:08.102281 12180 raft_consensus.cc:348] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b477eaea0824b39a4d919a695850b4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34577 } } peers { permanent_uuid: "2df36ca7a2634d1bbf049323c0061255" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } } peers { permanent_uuid: "0f689c7085f84ed29351ba6049816ae4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44623 } }
I20260929 20:55:08.102651 12180 raft_consensus.cc:374] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:55:08.102746 12180 raft_consensus.cc:729] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b477eaea0824b39a4d919a695850b4e, State: Initialized, Role: FOLLOWER
I20260929 20:55:08.103302 12180 consensus_queue.cc:260] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b477eaea0824b39a4d919a695850b4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34577 } } peers { permanent_uuid: "2df36ca7a2634d1bbf049323c0061255" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } } peers { permanent_uuid: "0f689c7085f84ed29351ba6049816ae4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44623 } }
I20260929 20:55:08.104974 12180 ts_tablet_manager.cc:1428] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 20:55:08.107321 12181 raft_consensus.cc:348] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b477eaea0824b39a4d919a695850b4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34577 } } peers { permanent_uuid: "2df36ca7a2634d1bbf049323c0061255" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } } peers { permanent_uuid: "0f689c7085f84ed29351ba6049816ae4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44623 } }
I20260929 20:55:08.107734 12181 raft_consensus.cc:374] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:55:08.107884 12181 raft_consensus.cc:729] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df36ca7a2634d1bbf049323c0061255, State: Initialized, Role: FOLLOWER
I20260929 20:55:08.108497 12181 consensus_queue.cc:260] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b477eaea0824b39a4d919a695850b4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34577 } } peers { permanent_uuid: "2df36ca7a2634d1bbf049323c0061255" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } } peers { permanent_uuid: "0f689c7085f84ed29351ba6049816ae4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44623 } }
I20260929 20:55:08.109714 12162 heartbeater.cc:503] Master 127.0.97.62:45219 was elected leader, sending a full tablet report...
I20260929 20:55:08.110378 12181 ts_tablet_manager.cc:1428] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260929 20:55:08.111004 12182 raft_consensus.cc:348] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b477eaea0824b39a4d919a695850b4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34577 } } peers { permanent_uuid: "2df36ca7a2634d1bbf049323c0061255" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } } peers { permanent_uuid: "0f689c7085f84ed29351ba6049816ae4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44623 } }
I20260929 20:55:08.111403 12182 raft_consensus.cc:374] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:55:08.111503 12182 raft_consensus.cc:729] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f689c7085f84ed29351ba6049816ae4, State: Initialized, Role: FOLLOWER
I20260929 20:55:08.112030 12182 consensus_queue.cc:260] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b477eaea0824b39a4d919a695850b4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34577 } } peers { permanent_uuid: "2df36ca7a2634d1bbf049323c0061255" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } } peers { permanent_uuid: "0f689c7085f84ed29351ba6049816ae4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44623 } }
I20260929 20:55:08.113044 12028 heartbeater.cc:503] Master 127.0.97.62:45219 was elected leader, sending a full tablet report...
I20260929 20:55:08.113571 12182 ts_tablet_manager.cc:1428] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4: Time spent starting tablet: real 0.013s user 0.000s sys 0.013s
I20260929 20:55:08.170163 12188 raft_consensus.cc:482] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:55:08.170517 12188 raft_consensus.cc:504] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b477eaea0824b39a4d919a695850b4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34577 } } peers { permanent_uuid: "2df36ca7a2634d1bbf049323c0061255" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } } peers { permanent_uuid: "0f689c7085f84ed29351ba6049816ae4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44623 } }
I20260929 20:55:08.172544 12188 leader_election.cc:288] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b477eaea0824b39a4d919a695850b4e (127.0.97.1:34577), 2df36ca7a2634d1bbf049323c0061255 (127.0.97.3:38993)
I20260929 20:55:08.179337 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:08.173161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56897 (local address 127.0.97.3:38993)
0929 20:55:08.173357 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:08.173363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:08.173833 (+ 470us) server_negotiation.cc:407] Connection header received
0929 20:55:08.173960 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:08.173963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:08.174027 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:08.174119 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:55:08.174840 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.175520 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.177505 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.178429 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.178524 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:08.178599 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:08.178953 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 20:55:08.179102 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:55:08.179415 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:08.172833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38993 (local address 127.0.97.2:56897)
0929 20:55:08.173712 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0929 20:55:08.173733 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:55:08.173858 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:55:08.174128 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:55:08.174132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:55:08.174171 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:55:08.174688 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:08.174701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.175690 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.175696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:08.177344 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:08.177356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.178633 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.178639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:08.178800 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:08.178835 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 20:55:08.178921 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 20:55:08.179055 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":53,"spinlock_wait_cycles":2304,"thread_start_us":462,"threads_started":1}
I20260929 20:55:08.179713 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:08.172883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47981 (local address 127.0.97.1:34577)
0929 20:55:08.173089 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:08.173096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:08.173775 (+ 679us) server_negotiation.cc:407] Connection header received
0929 20:55:08.173947 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:08.173953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:08.174024 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:08.174098 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:55:08.174756 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.176044 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.177894 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.178774 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.178912 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:08.179006 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:08.179351 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 20:55:08.179480 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:55:08.179800 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:08.172730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34577 (local address 127.0.97.2:47981)
0929 20:55:08.173659 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0929 20:55:08.173675 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:55:08.173802 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:55:08.174128 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:55:08.174133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:55:08.174161 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:55:08.174613 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:08.174623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.176156 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.176160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:08.177738 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:08.177751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.178994 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.179000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:08.179184 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:08.179218 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:55:08.179310 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 20:55:08.179442 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":51,"spinlock_wait_cycles":39552,"thread_start_us":744,"threads_started":1}
I20260929 20:55:08.180437 12117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761dfa13616241919fec671c6381a501" candidate_uuid: "0f689c7085f84ed29351ba6049816ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df36ca7a2634d1bbf049323c0061255" is_pre_election: true
I20260929 20:55:08.180685 11840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761dfa13616241919fec671c6381a501" candidate_uuid: "0f689c7085f84ed29351ba6049816ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b477eaea0824b39a4d919a695850b4e" is_pre_election: true
I20260929 20:55:08.180924 12117 raft_consensus.cc:2454] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f689c7085f84ed29351ba6049816ae4 in term 0.
I20260929 20:55:08.181140 11840 raft_consensus.cc:2454] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f689c7085f84ed29351ba6049816ae4 in term 0.
I20260929 20:55:08.181648 11918 leader_election.cc:302] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b477eaea0824b39a4d919a695850b4e, 0f689c7085f84ed29351ba6049816ae4, 2df36ca7a2634d1bbf049323c0061255; no voters:
I20260929 20:55:08.182029 12188 raft_consensus.cc:2781] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:55:08.182154 12188 raft_consensus.cc:482] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:55:08.182276 12188 raft_consensus.cc:3037] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:55:08.184401 12188 raft_consensus.cc:504] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b477eaea0824b39a4d919a695850b4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34577 } } peers { permanent_uuid: "2df36ca7a2634d1bbf049323c0061255" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } } peers { permanent_uuid: "0f689c7085f84ed29351ba6049816ae4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44623 } }
I20260929 20:55:08.184979 12188 leader_election.cc:288] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [CANDIDATE]: Term 1 election: Requested vote from peers 0b477eaea0824b39a4d919a695850b4e (127.0.97.1:34577), 2df36ca7a2634d1bbf049323c0061255 (127.0.97.3:38993)
I20260929 20:55:08.185436 11840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761dfa13616241919fec671c6381a501" candidate_uuid: "0f689c7085f84ed29351ba6049816ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b477eaea0824b39a4d919a695850b4e"
I20260929 20:55:08.185454 12117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761dfa13616241919fec671c6381a501" candidate_uuid: "0f689c7085f84ed29351ba6049816ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df36ca7a2634d1bbf049323c0061255"
I20260929 20:55:08.185623 12117 raft_consensus.cc:3037] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:55:08.185611 11840 raft_consensus.cc:3037] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:55:08.187733 12117 raft_consensus.cc:2454] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f689c7085f84ed29351ba6049816ae4 in term 1.
I20260929 20:55:08.188127 11840 raft_consensus.cc:2454] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f689c7085f84ed29351ba6049816ae4 in term 1.
I20260929 20:55:08.188127 11918 leader_election.cc:302] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f689c7085f84ed29351ba6049816ae4, 2df36ca7a2634d1bbf049323c0061255; no voters:
I20260929 20:55:08.188419 12188 raft_consensus.cc:2781] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:55:08.188838 12188 raft_consensus.cc:686] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [term 1 LEADER]: Becoming Leader. State: Replica: 0f689c7085f84ed29351ba6049816ae4, State: Running, Role: LEADER
I20260929 20:55:08.189375 12188 consensus_queue.cc:237] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b477eaea0824b39a4d919a695850b4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34577 } } peers { permanent_uuid: "2df36ca7a2634d1bbf049323c0061255" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } } peers { permanent_uuid: "0f689c7085f84ed29351ba6049816ae4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44623 } }
I20260929 20:55:08.193668 11671 catalog_manager.cc:5495] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f689c7085f84ed29351ba6049816ae4 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0f689c7085f84ed29351ba6049816ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b477eaea0824b39a4d919a695850b4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df36ca7a2634d1bbf049323c0061255" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f689c7085f84ed29351ba6049816ae4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44623 } health_report { overall_health: HEALTHY } } }
I20260929 20:55:08.213694 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:08.208372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57614 (local address 127.0.97.2:44623)
0929 20:55:08.208555 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:08.208561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:08.208606 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:55:08.208863 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:08.208868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:08.208990 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:08.209141 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:55:08.209507 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.210229 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.211601 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.211999 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.212101 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:08.212616 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:55:08.213123 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:55:08.213251 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:08.213333 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 20:55:08.213412 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 20:55:08.218264 12029 tablet.cc:2369] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:55:08.222257 12117 raft_consensus.cc:1261] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255 [term 1 FOLLOWER]: Refusing update from remote peer 0f689c7085f84ed29351ba6049816ae4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:55:08.222258 11840 raft_consensus.cc:1261] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e [term 1 FOLLOWER]: Refusing update from remote peer 0f689c7085f84ed29351ba6049816ae4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:55:08.223196 12191 consensus_queue.cc:1035] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df36ca7a2634d1bbf049323c0061255" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:55:08.223506 12188 consensus_queue.cc:1035] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b477eaea0824b39a4d919a695850b4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:55:08.228629 12163 tablet.cc:2369] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:55:08.238996 12195 mvcc.cc:205] Tried to move back new op lower bound from 7334769902468403200 to 7334769902348046336. Current Snapshot: MvccSnapshot[applied={T|T < 7334769902468403200 or (T in {7334769902468403200})}]
I20260929 20:55:08.239315 12194 mvcc.cc:205] Tried to move back new op lower bound from 7334769902468403200 to 7334769902348046336. Current Snapshot: MvccSnapshot[applied={T|T < 7334769902468403200 or (T in {7334769902468403200})}]
I20260929 20:55:08.251247 11671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:55:08.251492 11671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:55:08.255124 11671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93d96cc371d14e8a8810f33f2dc00681: Sending DeleteTablet for 3 replicas of tablet 761dfa13616241919fec671c6381a501
I20260929 20:55:08.256227 12097 tablet_service.cc:1514] Processing DeleteTablet for tablet 761dfa13616241919fec671c6381a501 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:55:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57544
I20260929 20:55:08.256264 11820 tablet_service.cc:1514] Processing DeleteTablet for tablet 761dfa13616241919fec671c6381a501 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:55:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46454
I20260929 20:55:08.256420 11963 tablet_service.cc:1514] Processing DeleteTablet for tablet 761dfa13616241919fec671c6381a501 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:55:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57606
I20260929 20:55:08.258772 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 11752
I20260929 20:55:08.269611 12204 tablet_replica.cc:330] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e: stopping tablet replica
I20260929 20:55:08.270577 12206 tablet_replica.cc:330] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4: stopping tablet replica
I20260929 20:55:08.270857 12204 raft_consensus.cc:2229] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:55:08.271320 12205 tablet_replica.cc:330] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255: stopping tablet replica
I20260929 20:55:08.271484 12204 raft_consensus.cc:2258] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:55:08.271942 12205 raft_consensus.cc:2229] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:55:08.272367 12206 raft_consensus.cc:2229] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:55:08.273396 12206 raft_consensus.cc:2258] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:55:08.273685 12205 raft_consensus.cc:2258] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:55:08.273862 12204 ts_tablet_manager.cc:1905] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:55:08.277942 12206 ts_tablet_manager.cc:1905] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:55:08.278415 12205 ts_tablet_manager.cc:1905] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 20:55:08 dist-test-slave-m9g1 krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:55:08.280825 12204 ts_tablet_manager.cc:1918] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:55:08.281025 12204 log.cc:1196] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/761dfa13616241919fec671c6381a501
I20260929 20:55:08.281586 12204 ts_tablet_manager.cc:1939] T 761dfa13616241919fec671c6381a501 P 0b477eaea0824b39a4d919a695850b4e: Deleting consensus metadata
I20260929 20:55:08.283718 11659 catalog_manager.cc:4841] TS 0b477eaea0824b39a4d919a695850b4e (127.0.97.1:34577): tablet 761dfa13616241919fec671c6381a501 (table test-table [id=26fb42d1fafc41358a47ff3490ad677a]) successfully deleted
I20260929 20:55:08.286144 12205 ts_tablet_manager.cc:1918] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:55:08.286410 12205 log.cc:1196] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/761dfa13616241919fec671c6381a501
I20260929 20:55:08.286960 12205 ts_tablet_manager.cc:1939] T 761dfa13616241919fec671c6381a501 P 2df36ca7a2634d1bbf049323c0061255: Deleting consensus metadata
I20260929 20:55:08.288622 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:08.260162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.0.97.1:34577)
0929 20:55:08.260349 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:08.260355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:08.265164 (+ 4809us) server_negotiation.cc:407] Connection header received
0929 20:55:08.265355 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:08.265361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:08.265495 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:08.265608 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:55:08.271886 (+ 6278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.273028 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.275022 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.275653 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.275757 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:08.281145 (+ 5388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:55:08.281164 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:55:08.281182 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:55:08.281236 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:55:08.285069 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:08.286031 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:08.286041 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:08.286044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:08.286244 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:08.286710 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:08.286715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:08.286719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:08.287156 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:55:08.287436 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:08.288029 (+ 593us) server_negotiation.cc:299] Negotiation successful
0929 20:55:08.288246 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:55:08.288866 11658 catalog_manager.cc:4841] TS 2df36ca7a2634d1bbf049323c0061255 (127.0.97.3:38993): tablet 761dfa13616241919fec671c6381a501 (table test-table [id=26fb42d1fafc41358a47ff3490ad677a]) successfully deleted
I20260929 20:55:08.290086 11880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:55:08.291293 12206 ts_tablet_manager.cc:1918] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:55:08.291509 12206 log.cc:1196] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/761dfa13616241919fec671c6381a501
I20260929 20:55:08.292012 12206 ts_tablet_manager.cc:1939] T 761dfa13616241919fec671c6381a501 P 0f689c7085f84ed29351ba6049816ae4: Deleting consensus metadata
I20260929 20:55:08.293845 11656 catalog_manager.cc:4841] TS 0f689c7085f84ed29351ba6049816ae4 (127.0.97.2:44623): tablet 761dfa13616241919fec671c6381a501 (table test-table [id=26fb42d1fafc41358a47ff3490ad677a]) successfully deleted
I20260929 20:55:08.440644 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 11752
I20260929 20:55:08.455176 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 11893
Sep 29 20:55:08 dist-test-slave-m9g1 krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:55:08.472863 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:08.455922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57626 (local address 127.0.97.2:44623)
0929 20:55:08.456086 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:08.456091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:08.456140 (+ 49us) server_negotiation.cc:407] Connection header received
0929 20:55:08.456334 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:08.456344 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:08.456411 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:08.456507 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:55:08.457778 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.458687 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.459913 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.460345 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.460462 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:08.464887 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:55:08.464901 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:55:08.464919 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:55:08.464970 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:55:08.469024 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:08.469978 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:08.469988 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:08.469992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:08.470159 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:08.470617 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:08.470623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:08.470627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:08.471072 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:55:08.471407 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:08.472251 (+ 844us) server_negotiation.cc:299] Negotiation successful
0929 20:55:08.472477 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:55:08.473654 12023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:55:08.586941 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 11893
I20260929 20:55:08.601223 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 12034
Sep 29 20:55:08 dist-test-slave-m9g1 krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:55:08.616348 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:08.601989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57556 (local address 127.0.97.3:38993)
0929 20:55:08.602155 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:08.602159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:08.602248 (+ 89us) server_negotiation.cc:407] Connection header received
0929 20:55:08.602433 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:08.602437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:08.602534 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:08.602618 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:55:08.603887 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.604621 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.605894 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:08.606343 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:08.606450 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:08.610746 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:55:08.610762 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:55:08.610778 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:55:08.610816 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:55:08.613519 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:08.614393 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:08.614402 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:08.614404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:08.614558 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:08.614975 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:08.614978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:08.614980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:08.615343 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:55:08.615566 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:08.615884 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 20:55:08.616051 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:55:08.617004 12157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:55:08.703756 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 12034
I20260929 20:55:08.716595 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 11640
I20260929 20:55:08.717772 11702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:55:08.827519 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 11640
I20260929 20:55:08.843766 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 20:55:09.178992 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 20:55:09.179212 388 mini_ranger.cc:66] Stopping Ranger...
I20260929 20:55:09.514369 388 mini_ranger.cc:68] Stopped Ranger
2026-09-29 20:55:09.514 UTC [11226] LOG: received smart shutdown request
2026-09-29 20:55:09.518 UTC [11226] LOG: background worker "logical replication launcher" (PID 11235) exited with exit code 1
2026-09-29 20:55:09.518 UTC [11230] LOG: shutting down
2026-09-29 20:55:09.676 UTC [11226] LOG: database system is shut down
2026-09-29T20:55:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48213 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:55:09 dist-test-slave-m9g1 krb5kdc[12220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:55:09 dist-test-slave-m9g1 krb5kdc[12220](info): set up 2 sockets
Sep 29 20:55:09 dist-test-slave-m9g1 krb5kdc[12220](info): commencing operation
krb5kdc: starting...
W20260929 20:55:10.790307 388 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:55:10 dist-test-slave-m9g1 krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:55:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:55:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260929 20:55:11.087249 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPL7Duu/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/postgres -l logfile start
2026-09-29 20:55:13.462 UTC [12250] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 20:55:13.462 UTC [12250] LOG: listening on IPv4 address "127.0.97.20", port 50597
2026-09-29 20:55:13.465 UTC [12250] LOG: listening on Unix socket "/tmp/.s.PGSQL.50597"
2026-09-29 20:55:13.485 UTC [12253] LOG: database system was shut down at 2026-09-29 20:55:12 UTC
2026-09-29 20:55:13.489 UTC [12250] LOG: database system is ready to accept connections
I20260929 20:55:14.365885 388 mini_postgres.cc:96] Postgres bound to 50597
2026-09-29 20:55:14.381 UTC [12261] FATAL: database "slave" does not exist
127.0.97.20:50597 - accepting connections
I20260929 20:55:14.381989 388 mini_ranger.cc:161] Starting Ranger...
I20260929 20:55:14.410559 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 20:55:14.880976 388 mini_ranger.cc:87] Created ranger Postgres database
I20260929 20:55:14.881076 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ranger-admin
2026-09-29 20:55:15,156 [I] DB FLAVOR :POSTGRES
2026-09-29 20:55:15,157 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:55:15,157 [I] Checking connection..
2026-09-29 20:55:15,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 20:55:15,395 [I] Checking connection passed.
2026-09-29 20:55:15,395 [I] --------- Verifying version history table ---------
2026-09-29 20:55:15,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:55:15,622 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:55:15,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:55:15,877 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:55:15,877 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 20:55:15,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 20:55:16.127 UTC [12325] WARNING: there is no transaction in progress
2026-09-29 20:55:16,133 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 20:55:16,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:55:16,381 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 20:55:16,381 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:55:16,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 20:55:16,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 20:55:16,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-m9g1.c.gcp-upstream.internal','N') ;"
2026-09-29 20:55:17,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:55:17,375 [I] Table x_portal_user does not exist in database ranger
2026-09-29 20:55:17,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:55:17,622 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 20:55:17,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:55:17,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:55:18,128 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 20:55:18,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 20:55:19.121 UTC [12433] WARNING: there is no transaction in progress
2026-09-29 20:55:19.140 UTC [12433] WARNING: there is no transaction in progress
2026-09-29 20:55:19.159 UTC [12433] WARNING: there is no transaction in progress
2026-09-29 20:55:19.177 UTC [12433] WARNING: there is no transaction in progress
2026-09-29 20:55:19.193 UTC [12433] WARNING: there is no transaction in progress
2026-09-29 20:55:19.210 UTC [12433] WARNING: there is no transaction in progress
2026-09-29 20:55:19.266 UTC [12433] WARNING: there is no transaction in progress
2026-09-29 20:55:19.273 UTC [12433] WARNING: there is no transaction in progress
2026-09-29 20:55:19.287 UTC [12433] WARNING: there is no transaction in progress
2026-09-29 20:55:19.298 UTC [12433] WARNING: there is no transaction in progress
2026-09-29 20:55:19.312 UTC [12433] WARNING: there is no transaction in progress
2026-09-29 20:55:19.329 UTC [12433] WARNING: there is no transaction in progress
2026-09-29 20:55:19.341 UTC [12433] WARNING: there is no transaction in progress
2026-09-29 20:55:19.348 UTC [12433] WARNING: there is no transaction in progress
2026-09-29 20:55:19.355 UTC [12433] WARNING: there is no transaction in progress
2026-09-29 20:55:19.362 UTC [12433] WARNING: there is no transaction in progress
2026-09-29 20:55:20,007 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 20:55:20,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 20:55:20,250 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 20:55:20,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:55:20,515 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 20:55:20,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:55:20,778 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 20:55:20,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:55:21,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:55:21,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-m9g1.c.gcp-upstream.internal';"
2026-09-29 20:55:21,482 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 20:55:21,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 20:55:21,744 [I] DB_PATCHES have already been applied
I20260929 20:55:21.750715 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPL7Duu/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 20:55:21.750813 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260929 20:55:21.764256 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:47453
Sep 29, 2026 8:55:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:55:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:55:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 8:55:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 8:55:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 8:55:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 8:55:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 20:55:22.795823 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.414s user 0.000s sys 0.043s
Sep 29, 2026 8:55:43 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 8:55:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 29, 2026 8:55:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 8:55:43 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 8:55:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 8:55:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 8:55:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 8:55:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 8:55:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 8:55:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 8:55:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 8:55:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 8:55:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 8:55:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 8:55:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 8:55:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 8:55:43 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 8:55:44 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 20:55:44.725050 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 20:55:44 dist-test-slave-m9g1 krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715344, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 20:55:44.888738 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 20:55:44.907753 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 20:55:45.826758 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 20:55:45.826877 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ranger-kms
I20260929 20:55:45.826926 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:50597
2026-09-29 20:55:46,158 [I] DB FLAVOR :POSTGRES
2026-09-29 20:55:46,158 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:55:46,158 [I] Checking connection
2026-09-29 20:55:46,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 20:55:46,394 [I] connection success
2026-09-29 20:55:46,394 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 20:55:46,395 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 20:55:46,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 20:55:46,632 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 20:55:46,632 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:55:46,633 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 20:55:46,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50597/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 20:55:46,879 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 20:55:47.347991 388 mini_ranger_kms.cc:310] Created kms service
I20260929 20:55:47.578732 388 mini_ranger_kms.cc:326] Created kudu user
I20260929 20:55:47.621536 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 20:55:47 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715344, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:55:48.005939 388 mini_ranger_kms.cc:384] Added ranger policy
I20260929 20:55:48.006309 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPL7Duu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPL7Duu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 20:55:48.006371 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260929 20:55:48.023870 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12613
I20260929 20:55:48.024008 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:49039
Sep 29, 2026 8:55:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:55:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:55:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 8:55:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 8:55:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 8:55:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 20:55:50.096302 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.208s user 0.063s sys 0.281s
I20260929 20:55:50.096540 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 20:55:50.096701 388 mini_ranger_kms.cc:401] 127.0.97.20:49039/kms/v1/keys
Sep 29 20:55:50 dist-test-slave-m9g1 krb5kdc[12220](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790715350, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 20:55:50 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715350, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 20:55:51 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715350, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 20:55:53 dist-test-slave-m9g1 krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:55:53.401237 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37055
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37055
--ranger_config_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49039/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:55:53.736394 12664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:55:53.736716 12664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:55:53.736804 12664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:55:53.744071 12664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:55:53.744199 12664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:55:53.744251 12664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:55:53.744302 12664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:55:53.744349 12664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:55:53.754781 12664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:49039/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37055
--ranger_config_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:55:53.756049 12664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:55:53.757352 12664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:55:53.769016 12669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:53.769069 12670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:53.769093 12672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:55:53.770010 12664 server_base.cc:1034] running on GCE node
I20260929 20:55:53.770825 12664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:55:53.772287 12664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:55:53.773485 12664 hybrid_clock.cc:648] HybridClock initialized: now 1790715353773437 us; error 39 us; skew 500 ppm
I20260929 20:55:53.773880 12664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:55:53 dist-test-slave-m9g1 krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:55:53.776974 12664 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:55:53.778570 12664 webserver.cc:460] Webserver started at http://127.0.97.62:44639/ using document root <none> and password file <none>
I20260929 20:55:53.779244 12664 fs_manager.cc:359] Metadata directory not provided
I20260929 20:55:53.779350 12664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:55:53.779646 12664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:55:53 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:55:53.832445 12664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "49a2a96c909e445dad819a9126e03162"
format_stamp: "Formatted at 2026-09-29 20:55:53 on dist-test-slave-m9g1"
server_key: "10d17a0357ec3d7f1623e878579e066f"
server_key_iv: "93e2464a32872756dd0f5fcabae237ec"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:53.833292 12664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "49a2a96c909e445dad819a9126e03162"
format_stamp: "Formatted at 2026-09-29 20:55:53 on dist-test-slave-m9g1"
server_key: "10d17a0357ec3d7f1623e878579e066f"
server_key_iv: "93e2464a32872756dd0f5fcabae237ec"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:53.838824 12664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 20:55:53.855641 12679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:53.857190 12664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:55:53.857367 12664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "49a2a96c909e445dad819a9126e03162"
format_stamp: "Formatted at 2026-09-29 20:55:53 on dist-test-slave-m9g1"
server_key: "10d17a0357ec3d7f1623e878579e066f"
server_key_iv: "93e2464a32872756dd0f5fcabae237ec"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:53.857589 12664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:55:53.880762 12664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:55:53.884788 12664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:55:53.885092 12664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:55:53.903208 12664 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37055
I20260929 20:55:53.903234 12730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37055 every 8 connection(s)
I20260929 20:55:53.904561 12664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:55:53.908362 12731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:55:53.913944 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 12664
I20260929 20:55:53.914130 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:55:53.914830 388 external_mini_cluster.cc:1428] Setting key 3afb50297dc617553c09c2527db42c45
I20260929 20:55:53.920421 12731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162: Bootstrap starting.
I20260929 20:55:53.924316 12731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162: Neither blocks nor log segments found. Creating new log.
I20260929 20:55:53.925302 12731 log.cc:824] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162: Log is configured to *not* fsync() on all Append() calls
I20260929 20:55:53.928145 12731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162: No bootstrap required, opened a new log
Sep 29 20:55:53 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:55:53.935987 12731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a2a96c909e445dad819a9126e03162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37055 } }
I20260929 20:55:53.936416 12731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:55:53.936537 12731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49a2a96c909e445dad819a9126e03162, State: Initialized, Role: FOLLOWER
I20260929 20:55:53.937175 12731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a2a96c909e445dad819a9126e03162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37055 } }
I20260929 20:55:53.937379 12731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:55:53.937505 12731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:55:53.937595 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:53.916843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50260 (local address 127.0.97.62:37055)
0929 20:55:53.917678 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:53.917696 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:53.917746 (+ 50us) server_negotiation.cc:407] Connection header received
0929 20:55:53.920034 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:53.920063 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:53.920596 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:53.921048 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:55:53.922148 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:53.924119 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:53.925452 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:53.926398 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:53.926587 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:53.930349 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:55:53.930360 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:55:53.930376 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:55:53.930418 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:55:53.933431 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:53.934125 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:53.934132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:53.934142 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:53.934356 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:53.934734 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:53.934739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:53.934745 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:53.935328 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:55:53.935601 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:53.936065 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 20:55:53.936485 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":352,"threads_started":1}
I20260929 20:55:53.937681 12731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:55:53.940100 12731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a2a96c909e445dad819a9126e03162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37055 } }
I20260929 20:55:53.940639 12731 leader_election.cc:302] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49a2a96c909e445dad819a9126e03162; no voters:
I20260929 20:55:53.941217 12731 leader_election.cc:288] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:55:53.941344 12736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:55:53.941944 12736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [term 1 LEADER]: Becoming Leader. State: Replica: 49a2a96c909e445dad819a9126e03162, State: Running, Role: LEADER
I20260929 20:55:53.942575 12736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a2a96c909e445dad819a9126e03162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37055 } }
I20260929 20:55:53.943216 12731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:55:53.945207 12738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49a2a96c909e445dad819a9126e03162. Latest consensus state: current_term: 1 leader_uuid: "49a2a96c909e445dad819a9126e03162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a2a96c909e445dad819a9126e03162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37055 } } }
I20260929 20:55:53.945602 12738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [sys.catalog]: This master's current role is: LEADER
I20260929 20:55:53.946105 12737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49a2a96c909e445dad819a9126e03162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a2a96c909e445dad819a9126e03162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37055 } } }
I20260929 20:55:53.946517 12737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [sys.catalog]: This master's current role is: LEADER
I20260929 20:55:53.951447 12744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:55:53.952232 12731 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 20:55:53.964320 12744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:55:53.980937 12744 catalog_manager.cc:1347] Generated new cluster ID: a14568b32874456e9a5b0b9554cd2cca
I20260929 20:55:53.981070 12744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:55:54.000955 12744 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:55:54.002980 12744 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:55:54.017094 12744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162: Generated new TSK 0
I20260929 20:55:54.018338 12744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:55:54.752175 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49039/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37055
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:55:55 dist-test-slave-m9g1 krb5kdc[12220](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790715355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 20:55:55.166827 12776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:55:55.167235 12776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:55:55.167397 12776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:55:55.179208 12776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:55:55.179346 12776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:55:55.179586 12776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:55:55.195492 12776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:49039/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:55:55.197358 12776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:55:55.199095 12776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:55:55.220784 12786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:55.220861 12787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:55.236254 12789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:55:55.236696 12776 server_base.cc:1034] running on GCE node
I20260929 20:55:55.237689 12776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:55:55.239033 12776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:55:55.240346 12776 hybrid_clock.cc:648] HybridClock initialized: now 1790715355240274 us; error 52 us; skew 500 ppm
I20260929 20:55:55.240713 12776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:55:55 dist-test-slave-m9g1 krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:55:55.244727 12776 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:55:55.246549 12776 webserver.cc:460] Webserver started at http://127.0.97.1:38395/ using document root <none> and password file <none>
I20260929 20:55:55.247498 12776 fs_manager.cc:359] Metadata directory not provided
I20260929 20:55:55.247628 12776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:55:55.247968 12776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:55:55 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:55:55.266726 12776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "81d118f2752749a2b6497c8d5fe23a7e"
format_stamp: "Formatted at 2026-09-29 20:55:55 on dist-test-slave-m9g1"
server_key: "0196df9ca2be59971e4e7e94b79c9dcb"
server_key_iv: "2090266ce4c874dbba7e408a16581823"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:55.267987 12776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "81d118f2752749a2b6497c8d5fe23a7e"
format_stamp: "Formatted at 2026-09-29 20:55:55 on dist-test-slave-m9g1"
server_key: "0196df9ca2be59971e4e7e94b79c9dcb"
server_key_iv: "2090266ce4c874dbba7e408a16581823"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:55.283176 12776 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.005s sys 0.008s
I20260929 20:55:55.318382 12796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:55.320647 12776 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.010s sys 0.008s
I20260929 20:55:55.320977 12776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "81d118f2752749a2b6497c8d5fe23a7e"
format_stamp: "Formatted at 2026-09-29 20:55:55 on dist-test-slave-m9g1"
server_key: "0196df9ca2be59971e4e7e94b79c9dcb"
server_key_iv: "2090266ce4c874dbba7e408a16581823"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:55.321373 12776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:55:55.371776 12776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:55:55.377445 12776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:55:55.377837 12776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:55:55.385620 12776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:55:55.394475 12776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:55:55.394662 12776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:55.394876 12776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:55:55.395004 12776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:55:55 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:55:55.844832 12776 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44747
I20260929 20:55:55.844729 12909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44747 every 8 connection(s)
I20260929 20:55:55.846303 12776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
Sep 29 20:55:55 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:55:55.852353 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 12776
I20260929 20:55:55.852563 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:55:55.853451 388 external_mini_cluster.cc:1428] Setting key 2bbcf5b6889473bd346454be9db6b7e1
Sep 29 20:55:55 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:55:55.887548 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:55.851630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51231 (local address 127.0.97.62:37055)
0929 20:55:55.854565 (+ 2935us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:55.854571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:55.855336 (+ 765us) server_negotiation.cc:407] Connection header received
0929 20:55:55.858082 (+ 2746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:55.858089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:55.858178 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:55.858281 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:55:55.860625 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:55.861728 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:55.864609 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:55.865181 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:55.865283 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:55.869628 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:55:55.869635 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:55:55.869640 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:55:55.869683 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:55:55.873155 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:55.884388 (+ 11233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:55.884395 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:55.884399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:55.884599 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:55.885233 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:55.885239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:55.885242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:55.885632 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:55:55.885773 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:55.886884 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0929 20:55:55.887143 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2795,"spinlock_wait_cycles":15744,"thread_start_us":2751,"threads_started":1}
I20260929 20:55:55.890127 12913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:55.852110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37055 (local address 127.0.97.1:51231)
0929 20:55:55.855145 (+ 3035us) negotiation.cc:107] Waiting for socket to connect
0929 20:55:55.855202 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 20:55:55.857652 (+ 2450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:55:55.858857 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:55:55.858870 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:55:55.859523 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:55:55.860394 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:55.860422 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:55.862533 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:55.862539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:55.864438 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:55.864453 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:55.865630 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:55.865636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:55.865914 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:55.866738 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:55:55.866769 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:55:55.869325 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:55:55.883388 (+ 14063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:55:55.883400 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:55:55.883421 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:55:55.884186 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:55:55.884776 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:55:55.884781 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:55:55.884785 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:55:55.885089 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:55:55.885892 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:55:55.885900 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:55:55.886380 (+ 480us) client_negotiation.cc:769] Sending connection context
0929 20:55:55.888050 (+ 1670us) client_negotiation.cc:240] Negotiation successful
0929 20:55:55.888663 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2693,"thread_start_us":8327,"threads_started":1}
I20260929 20:55:55.892809 12910 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37055
I20260929 20:55:55.893179 12910 heartbeater.cc:464] Registering TS with master...
I20260929 20:55:55.894153 12910 heartbeater.cc:511] Master 127.0.97.62:37055 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:55:55.896656 12696 ts_manager.cc:194] Registered new tserver with Master: 81d118f2752749a2b6497c8d5fe23a7e (127.0.97.1:44747)
I20260929 20:55:55.898366 12696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:55:55.945667 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49039/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37055
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:55:56.323118 12917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:55:56.323427 12917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:55:56.323546 12917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:55:56.331298 12917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:55:56.331444 12917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:55:56.331590 12917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:55:56.341711 12917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:49039/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:55:56.343089 12917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:55:56.344465 12917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:55:56.359230 12929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:56.359658 12932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:56.359758 12930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:55:56.360806 12917 server_base.cc:1034] running on GCE node
I20260929 20:55:56.361588 12917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:55:56.362811 12917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:55:56.364032 12917 hybrid_clock.cc:648] HybridClock initialized: now 1790715356363957 us; error 63 us; skew 500 ppm
I20260929 20:55:56.364418 12917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:55:56 dist-test-slave-m9g1 krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:55:56.368347 12917 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:55:56.370298 12917 webserver.cc:460] Webserver started at http://127.0.97.2:41141/ using document root <none> and password file <none>
I20260929 20:55:56.371191 12917 fs_manager.cc:359] Metadata directory not provided
I20260929 20:55:56.371325 12917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:55:56.371667 12917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:55:56 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:55:56.390041 12917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "bd796e8c39d74bd198de2e53f5dcb6ff"
format_stamp: "Formatted at 2026-09-29 20:55:56 on dist-test-slave-m9g1"
server_key: "b19ed4eefd796e96d803ccb29f3fc5b7"
server_key_iv: "201d8b7aaf6d4f5d8c73122330964c44"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:56.391228 12917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bd796e8c39d74bd198de2e53f5dcb6ff"
format_stamp: "Formatted at 2026-09-29 20:55:56 on dist-test-slave-m9g1"
server_key: "b19ed4eefd796e96d803ccb29f3fc5b7"
server_key_iv: "201d8b7aaf6d4f5d8c73122330964c44"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:56.398272 12917 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 20:55:56.418845 12939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:56.420859 12917 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.003s
I20260929 20:55:56.421064 12917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "bd796e8c39d74bd198de2e53f5dcb6ff"
format_stamp: "Formatted at 2026-09-29 20:55:56 on dist-test-slave-m9g1"
server_key: "b19ed4eefd796e96d803ccb29f3fc5b7"
server_key_iv: "201d8b7aaf6d4f5d8c73122330964c44"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:56.421327 12917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:55:56.456337 12917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:55:56.461680 12917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:55:56.462015 12917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:55:56.463207 12917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:55:56.464948 12917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:55:56.465035 12917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:56.465201 12917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:55:56.465261 12917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:56.507956 12917 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39483
I20260929 20:55:56.508016 13051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39483 every 8 connection(s)
I20260929 20:55:56.509641 12917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:55:56.511672 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 12917
I20260929 20:55:56.511838 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:55:56.512346 388 external_mini_cluster.cc:1428] Setting key 9bb4fec4d75344bcf229e698b515ef9d
Sep 29 20:55:56 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 20:55:56.556602 13054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:56.514929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50839 (local address 127.0.97.62:37055)
0929 20:55:56.530080 (+ 15151us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:56.530087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:56.533364 (+ 3277us) server_negotiation.cc:407] Connection header received
0929 20:55:56.534596 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:56.534604 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:56.534700 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:56.534778 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:55:56.537068 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:56.538246 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:56.540677 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:56.541327 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:56.541429 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:56.546053 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:55:56.546060 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:55:56.546065 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:55:56.546107 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:55:56.549489 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:56.550352 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:56.550357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:56.550359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:56.550490 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:56.550936 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:56.550939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:56.550941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:56.551257 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:55:56.551406 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:56.555976 (+ 4570us) server_negotiation.cc:299] Negotiation successful
0929 20:55:56.556247 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14986,"thread_start_us":15161,"threads_started":1}
I20260929 20:55:56.558985 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:56.515680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37055 (local address 127.0.97.2:50839)
0929 20:55:56.531367 (+ 15687us) negotiation.cc:107] Waiting for socket to connect
0929 20:55:56.531428 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 20:55:56.534065 (+ 2637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:55:56.535370 (+ 1305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:55:56.535383 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:55:56.536011 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:55:56.536839 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:56.536863 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:56.538465 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:56.538471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:56.540501 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:56.540516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:56.541786 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:56.541793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:56.542108 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:56.542966 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:55:56.542996 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:55:56.545757 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:55:56.549680 (+ 3923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:55:56.549691 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:55:56.549712 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:55:56.550166 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:55:56.550602 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:55:56.550608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:55:56.550612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:55:56.550814 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:55:56.551382 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:55:56.551390 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:55:56.555458 (+ 4068us) client_negotiation.cc:769] Sending connection context
0929 20:55:56.556951 (+ 1493us) client_negotiation.cc:240] Negotiation successful
0929 20:55:56.557528 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15365,"thread_start_us":15168,"threads_started":1}
I20260929 20:55:56.560997 13052 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37055
I20260929 20:55:56.561321 13052 heartbeater.cc:464] Registering TS with master...
I20260929 20:55:56.562202 13052 heartbeater.cc:511] Master 127.0.97.62:37055 requested a full tablet report, sending...
I20260929 20:55:56.564003 12696 ts_manager.cc:194] Registered new tserver with Master: bd796e8c39d74bd198de2e53f5dcb6ff (127.0.97.2:39483)
I20260929 20:55:56.565073 12696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:55:56.585037 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49039/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37055
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:55:56.902065 12910 heartbeater.cc:503] Master 127.0.97.62:37055 was elected leader, sending a full tablet report...
W20260929 20:55:56.925547 13058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:55:56.925841 13058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:55:56.925995 13058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:55:56.933590 13058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:55:56.933730 13058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:55:56.933878 13058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:55:56.943889 13058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:49039/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:55:56.945266 13058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:55:56.946745 13058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:55:56.959564 13063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:56.959547 13064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:56.959631 13066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:55:56.960217 13058 server_base.cc:1034] running on GCE node
I20260929 20:55:56.960856 13058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:55:56.961856 13058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:55:56.963071 13058 hybrid_clock.cc:648] HybridClock initialized: now 1790715356963011 us; error 53 us; skew 500 ppm
I20260929 20:55:56.963354 13058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:55:56 dist-test-slave-m9g1 krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:55:56.966522 13058 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:55:56.967940 13058 webserver.cc:460] Webserver started at http://127.0.97.3:34275/ using document root <none> and password file <none>
I20260929 20:55:56.968606 13058 fs_manager.cc:359] Metadata directory not provided
I20260929 20:55:56.968708 13058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:55:56.969002 13058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:55:56 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:55:56.983860 13058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "7a3f0d6f70e1425b8a4d423ecc064801"
format_stamp: "Formatted at 2026-09-29 20:55:56 on dist-test-slave-m9g1"
server_key: "ead1641a28e6e2044cf23f33acca3658"
server_key_iv: "5ee11b3fdbe70b52f4057b2d1a82f428"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:56.985021 13058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7a3f0d6f70e1425b8a4d423ecc064801"
format_stamp: "Formatted at 2026-09-29 20:55:56 on dist-test-slave-m9g1"
server_key: "ead1641a28e6e2044cf23f33acca3658"
server_key_iv: "5ee11b3fdbe70b52f4057b2d1a82f428"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:56.992223 13058 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260929 20:55:57.007840 13073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:57.009541 13058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:55:57.009727 13058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "7a3f0d6f70e1425b8a4d423ecc064801"
format_stamp: "Formatted at 2026-09-29 20:55:56 on dist-test-slave-m9g1"
server_key: "ead1641a28e6e2044cf23f33acca3658"
server_key_iv: "5ee11b3fdbe70b52f4057b2d1a82f428"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:57.009941 13058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:55:57.032618 13058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:55:57.036767 13058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:55:57.037057 13058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:55:57.037986 13058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:55:57.039403 13058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:55:57.039480 13058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:57.039582 13058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:55:57.039623 13058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:57.069245 13058 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42893
I20260929 20:55:57.069267 13185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42893 every 8 connection(s)
I20260929 20:55:57.070587 13058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:55:57.077898 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 13058
I20260929 20:55:57.078069 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:55:57.078539 388 external_mini_cluster.cc:1428] Setting key c0fb4e3002ccc82e66d8151986e01c72
Sep 29 20:55:57 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:55:57.090026 13188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:57.073112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34219 (local address 127.0.97.62:37055)
0929 20:55:57.073579 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:57.073584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:57.074452 (+ 868us) server_negotiation.cc:407] Connection header received
0929 20:55:57.076291 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:57.076297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:57.076401 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:57.076497 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:55:57.078055 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:57.078878 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:57.080692 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:57.081154 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:57.081260 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:57.084481 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:55:57.084486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:55:57.084490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:55:57.084523 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:55:57.087061 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:57.087772 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:57.087776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:57.087779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:57.087905 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:57.088330 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:57.088334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:57.088336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:57.088612 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:55:57.088750 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:57.089423 (+ 673us) server_negotiation.cc:299] Negotiation successful
0929 20:55:57.089666 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":238,"threads_started":1}
I20260929 20:55:57.090988 13189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:57.073557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37055 (local address 127.0.97.3:34219)
0929 20:55:57.074292 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 20:55:57.074338 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:55:57.076001 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:55:57.076764 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:55:57.076774 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:55:57.077207 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:55:57.077870 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:57.077890 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:57.079028 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:57.079032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:57.080568 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:57.080583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:57.081373 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:57.081380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:57.081622 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:57.082288 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:55:57.082310 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:55:57.084266 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:55:57.087193 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:55:57.087203 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:55:57.087223 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:55:57.087658 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:55:57.088010 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:55:57.088015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:55:57.088019 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:55:57.088224 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:55:57.088732 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:55:57.088740 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:55:57.089074 (+ 334us) client_negotiation.cc:769] Sending connection context
0929 20:55:57.089318 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 20:55:57.089838 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":257,"threads_started":1}
I20260929 20:55:57.092453 13186 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37055
I20260929 20:55:57.092733 13186 heartbeater.cc:464] Registering TS with master...
I20260929 20:55:57.093420 13186 heartbeater.cc:511] Master 127.0.97.62:37055 requested a full tablet report, sending...
I20260929 20:55:57.094830 12696 ts_manager.cc:194] Registered new tserver with Master: 7a3f0d6f70e1425b8a4d423ecc064801 (127.0.97.3:42893)
I20260929 20:55:57.095633 12696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34219
I20260929 20:55:57.106411 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:55:57.108081 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 12776
Sep 29 20:55:57 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:55:57.123653 13191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:57.109064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49046 (local address 127.0.97.1:44747)
0929 20:55:57.109633 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:57.109640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:57.109666 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:55:57.109727 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:57.109733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:57.109935 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:57.110068 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:55:57.111494 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:57.112537 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:57.113990 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:57.114687 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:57.114776 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:57.118711 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:55:57.118723 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:55:57.118739 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:55:57.118774 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:55:57.121255 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:57.121921 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:57.121929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:57.121931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:57.122074 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:57.122411 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:57.122414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:57.122416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:57.122744 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:55:57.122981 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:57.123179 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 20:55:57.123365 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":291,"threads_started":1}
I20260929 20:55:57.124912 12905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:55:57.206730 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 12776
I20260929 20:55:57.219910 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 12917
Sep 29 20:55:57 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:55:57.236001 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:57.220760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51074 (local address 127.0.97.2:39483)
0929 20:55:57.221247 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:57.221255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:57.221281 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:55:57.221340 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:57.221346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:57.221563 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:57.221714 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:55:57.223020 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:57.224017 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:57.225308 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:57.225856 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:57.225970 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:57.230288 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:55:57.230298 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:55:57.230314 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:55:57.230352 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:55:57.233079 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:57.233819 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:57.233826 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:57.233829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:57.233966 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:57.234381 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:57.234386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:57.234388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:57.234744 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:55:57.235086 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:57.235396 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 20:55:57.235670 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":230,"threads_started":1}
I20260929 20:55:57.237228 13047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:55:57.319584 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 12917
I20260929 20:55:57.331892 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 13058
Sep 29 20:55:57 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:55:57.347616 13195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:57.332777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48608 (local address 127.0.97.3:42893)
0929 20:55:57.333344 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:57.333351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:57.333379 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:55:57.333447 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:57.333454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:57.333688 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:57.333817 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:55:57.335090 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:57.336145 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:57.337449 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:57.338003 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:57.338100 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:57.342150 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:55:57.342164 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:55:57.342180 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:55:57.342262 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:55:57.344981 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:57.345681 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:57.345688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:57.345691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:57.345822 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:57.346093 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:57.346096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:57.346098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:57.346486 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:55:57.346723 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:57.347037 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 20:55:57.347239 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":283,"threads_started":1}
I20260929 20:55:57.348851 13181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:55:57.431830 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 13058
I20260929 20:55:57.444324 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 12664
I20260929 20:55:57.445524 12726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:55:57.551869 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 12664
I20260929 20:55:57.569705 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37055
--webserver_interface=127.0.97.62
--webserver_port=44639
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37055
--ranger_config_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49039/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:49039/kms with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:55:57.890089 13198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:55:57.890429 13198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:55:57.890517 13198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:55:57.897866 13198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:55:57.898005 13198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:55:57.898052 13198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:55:57.898108 13198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:55:57.898155 13198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:55:57.908615 13198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:49039/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37055
--ranger_config_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:55:57.909909 13198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:55:57.911275 13198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:55:57.922987 13203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:57.922963 13204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:57.923200 13206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:55:57.923697 13198 server_base.cc:1034] running on GCE node
I20260929 20:55:57.924419 13198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:55:57.925838 13198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:55:57.927055 13198 hybrid_clock.cc:648] HybridClock initialized: now 1790715357927001 us; error 44 us; skew 500 ppm
I20260929 20:55:57.927415 13198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:55:57 dist-test-slave-m9g1 krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:55:57.930518 13198 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:55:57.931978 13198 webserver.cc:460] Webserver started at http://127.0.97.62:44639/ using document root <none> and password file <none>
I20260929 20:55:57.932647 13198 fs_manager.cc:359] Metadata directory not provided
I20260929 20:55:57.932753 13198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:55:57 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:55:57.983681 13198 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:55:57.986780 13214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:57.988279 13198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:55:57.988462 13198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "49a2a96c909e445dad819a9126e03162"
format_stamp: "Formatted at 2026-09-29 20:55:53 on dist-test-slave-m9g1"
server_key: "10d17a0357ec3d7f1623e878579e066f"
server_key_iv: "93e2464a32872756dd0f5fcabae237ec"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:57.989487 13198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:55:58.009728 13198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:55:58.013833 13198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:55:58.014127 13198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:55:58.031723 13198 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37055
I20260929 20:55:58.031744 13265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37055 every 8 connection(s)
I20260929 20:55:58.033241 13198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:55:58.040190 13266 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:55:58.042728 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 13198
I20260929 20:55:58.043747 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49039/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44747
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37055
--webserver_port=38395
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:49039/kms with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:55:58.044093 13266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162: Bootstrap starting.
I20260929 20:55:58.064805 13266 log.cc:824] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162: Log is configured to *not* fsync() on all Append() calls
I20260929 20:55:58.074041 13266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:55:58.074599 13266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162: Bootstrap complete.
I20260929 20:55:58.083353 13266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a2a96c909e445dad819a9126e03162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37055 } }
I20260929 20:55:58.084182 13266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49a2a96c909e445dad819a9126e03162, State: Initialized, Role: FOLLOWER
I20260929 20:55:58.084784 13266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a2a96c909e445dad819a9126e03162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37055 } }
I20260929 20:55:58.084990 13266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:55:58.085094 13266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:55:58.085249 13266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:55:58.089075 13266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a2a96c909e445dad819a9126e03162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37055 } }
I20260929 20:55:58.089553 13266 leader_election.cc:302] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49a2a96c909e445dad819a9126e03162; no voters:
I20260929 20:55:58.090143 13266 leader_election.cc:288] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:55:58.090303 13270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:55:58.091285 13270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [term 2 LEADER]: Becoming Leader. State: Replica: 49a2a96c909e445dad819a9126e03162, State: Running, Role: LEADER
I20260929 20:55:58.091833 13270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a2a96c909e445dad819a9126e03162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37055 } }
I20260929 20:55:58.092235 13266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:55:58.094565 13272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49a2a96c909e445dad819a9126e03162. Latest consensus state: current_term: 2 leader_uuid: "49a2a96c909e445dad819a9126e03162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a2a96c909e445dad819a9126e03162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37055 } } }
I20260929 20:55:58.094977 13272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [sys.catalog]: This master's current role is: LEADER
I20260929 20:55:58.095515 13271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "49a2a96c909e445dad819a9126e03162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a2a96c909e445dad819a9126e03162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37055 } } }
I20260929 20:55:58.095799 13271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162 [sys.catalog]: This master's current role is: LEADER
I20260929 20:55:58.105841 13276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:55:58.111613 13276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:55:58.112982 13276 catalog_manager.cc:1259] Loaded cluster ID: a14568b32874456e9a5b0b9554cd2cca
I20260929 20:55:58.113085 13276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:55:58.117236 13276 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:55:58.118770 13276 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162: Loaded TSK: 0
I20260929 20:55:58.120543 13276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:55:58.123566 13266 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260929 20:55:58.461270 13268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:55:58.461678 13268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:55:58.461835 13268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:55:58.473898 13268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:55:58.474078 13268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:55:58.474318 13268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:55:58.489876 13268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:49039/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38395
--tserver_master_addrs=127.0.97.62:37055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:55:58.491303 13268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:55:58.492859 13268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:55:58.527194 13303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:58.527482 13304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:58.532496 13306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:55:58.532804 13268 server_base.cc:1034] running on GCE node
I20260929 20:55:58.533710 13268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:55:58.535096 13268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:55:58.538467 13268 hybrid_clock.cc:648] HybridClock initialized: now 1790715358538228 us; error 219 us; skew 500 ppm
I20260929 20:55:58.538973 13268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:55:58 dist-test-slave-m9g1 krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:55:58.543576 13268 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:55:58.551532 13268 webserver.cc:460] Webserver started at http://127.0.97.1:38395/ using document root <none> and password file <none>
I20260929 20:55:58.552690 13268 fs_manager.cc:359] Metadata directory not provided
I20260929 20:55:58.552917 13268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:55:58 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:55:58.593122 13268 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.007s sys 0.008s
I20260929 20:55:58.598866 13314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:58.601810 13268 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.004s
I20260929 20:55:58.602048 13268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "81d118f2752749a2b6497c8d5fe23a7e"
format_stamp: "Formatted at 2026-09-29 20:55:55 on dist-test-slave-m9g1"
server_key: "0196df9ca2be59971e4e7e94b79c9dcb"
server_key_iv: "2090266ce4c874dbba7e408a16581823"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:58.603610 13268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:55:58.633821 13268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:55:58.639433 13268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:55:58.639775 13268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:55:58.641593 13268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:55:58.646031 13268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:55:58.646157 13268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:58.646298 13268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:55:58.646368 13268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:58.718328 13268 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44747
I20260929 20:55:58.718375 13427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44747 every 8 connection(s)
I20260929 20:55:58.719937 13268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:55:58.729296 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 13268
I20260929 20:55:58.730172 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49039/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:39483
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37055
--webserver_port=41141
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:49039/kms with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:55:58 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:55:58.784300 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:58.724854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43589 (local address 127.0.97.62:37055)
0929 20:55:58.752145 (+ 27291us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:58.752170 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:58.755414 (+ 3244us) server_negotiation.cc:407] Connection header received
0929 20:55:58.755816 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:58.755831 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:58.756255 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:58.756614 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:55:58.758673 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:58.760759 (+ 2086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:58.763351 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:58.764527 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:58.764740 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:58.773384 (+ 8644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:55:58.773397 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:55:58.773420 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:55:58.773478 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:55:58.778598 (+ 5120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:58.779761 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:58.779771 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:58.779785 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:58.780014 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:58.780601 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:58.780606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:58.780609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:58.781253 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:55:58.781449 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:58.782504 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0929 20:55:58.782964 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26986,"thread_start_us":26749,"threads_started":1}
I20260929 20:55:58.786633 13430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:58.723615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37055 (local address 127.0.97.1:43589)
0929 20:55:58.752805 (+ 29190us) negotiation.cc:107] Waiting for socket to connect
0929 20:55:58.752869 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 20:55:58.754943 (+ 2074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:55:58.756816 (+ 1873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:55:58.756827 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:55:58.757457 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:55:58.758353 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:58.758383 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:58.760986 (+ 2603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:58.760992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:58.763143 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:58.763171 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:58.765138 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:58.765146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:58.765477 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:58.770015 (+ 4538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:55:58.770062 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:55:58.772964 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:55:58.778846 (+ 5882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:55:58.778855 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:55:58.778872 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:55:58.779547 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:55:58.780178 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:55:58.780183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:55:58.780186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:55:58.780454 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:55:58.781524 (+ 1070us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:55:58.781532 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:55:58.781917 (+ 385us) client_negotiation.cc:769] Sending connection context
0929 20:55:58.784777 (+ 2860us) client_negotiation.cc:240] Negotiation successful
0929 20:55:58.785370 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28873,"thread_start_us":28636,"threads_started":1}
I20260929 20:55:58.791323 13428 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37055
I20260929 20:55:58.791675 13428 heartbeater.cc:464] Registering TS with master...
I20260929 20:55:58.792618 13428 heartbeater.cc:511] Master 127.0.97.62:37055 requested a full tablet report, sending...
W20260929 20:55:58.796152 13428 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37055 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 20:55:58.796396 13428 heartbeater.cc:464] Registering TS with master...
I20260929 20:55:58.796720 13428 heartbeater.cc:511] Master 127.0.97.62:37055 requested a full tablet report, sending...
I20260929 20:55:58.798132 13428 heartbeater.cc:464] Registering TS with master...
I20260929 20:55:58.798699 13428 heartbeater.cc:511] Master 127.0.97.62:37055 requested a full tablet report, sending...
W20260929 20:55:58.800004 13428 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:55:59.171402 13432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:55:59.171741 13432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:55:59.171873 13432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:55:59.179517 13432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:55:59.179646 13432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:55:59.179798 13432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:55:59.190980 13432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:49039/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:39483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=41141
--tserver_master_addrs=127.0.97.62:37055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:55:59.192938 13432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:55:59.194823 13432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:55:59.211156 13446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:59.219677 13447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:59.220973 13449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:55:59.221551 13432 server_base.cc:1034] running on GCE node
I20260929 20:55:59.222250 13432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:55:59.223352 13432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:55:59.224604 13432 hybrid_clock.cc:648] HybridClock initialized: now 1790715359224541 us; error 55 us; skew 500 ppm
I20260929 20:55:59.224931 13432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:55:59 dist-test-slave-m9g1 krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:55:59.229141 13432 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:55:59.231282 13432 webserver.cc:460] Webserver started at http://127.0.97.2:41141/ using document root <none> and password file <none>
I20260929 20:55:59.232271 13432 fs_manager.cc:359] Metadata directory not provided
I20260929 20:55:59.232429 13432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:55:59 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:55:59.271322 13432 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.008s sys 0.007s
I20260929 20:55:59.286986 13457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:55:59.289322 13432 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.011s sys 0.004s
I20260929 20:55:59.289646 13432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "bd796e8c39d74bd198de2e53f5dcb6ff"
format_stamp: "Formatted at 2026-09-29 20:55:56 on dist-test-slave-m9g1"
server_key: "b19ed4eefd796e96d803ccb29f3fc5b7"
server_key_iv: "201d8b7aaf6d4f5d8c73122330964c44"
server_key_version: "kuduclusterkey@0"
I20260929 20:55:59.291353 13432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:55:59.334370 13432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:55:59.339831 13432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:55:59.340345 13432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:55:59.341768 13432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:55:59.355155 13432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:55:59.355393 13432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 20:55:59.355544 13432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:55:59.355645 13432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:55:59 dist-test-slave-m9g1 krb5kdc[12220](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790715359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:55:59.473631 13432 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39483
I20260929 20:55:59.473695 13571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39483 every 8 connection(s)
I20260929 20:55:59.475754 13432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:55:59.479630 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 13432
I20260929 20:55:59.480382 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49039/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42893
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37055
--webserver_port=34275
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:49039/kms with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:55:59 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:55:59.507897 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:59.479982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47795 (local address 127.0.97.62:37055)
0929 20:55:59.484616 (+ 4634us) server_negotiation.cc:206] Beginning negotiation
0929 20:55:59.484622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:55:59.484646 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:55:59.484727 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:55:59.484733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:55:59.484837 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:55:59.484960 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:55:59.487107 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:59.488286 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:59.490943 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:55:59.491576 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:55:59.491721 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:59.500023 (+ 8302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:55:59.500032 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:55:59.500038 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:55:59.500085 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:55:59.503722 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:59.504878 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:59.504885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:59.504889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:59.505076 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:55:59.505674 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:55:59.505680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:55:59.505683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:55:59.506074 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:55:59.506264 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 20:55:59.507203 (+ 939us) server_negotiation.cc:299] Negotiation successful
0929 20:55:59.507479 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4495,"thread_start_us":4413,"threads_started":1}
I20260929 20:55:59.509526 13574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:55:59.479318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37055 (local address 127.0.97.2:47795)
0929 20:55:59.480606 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0929 20:55:59.480666 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 20:55:59.483227 (+ 2561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:55:59.485301 (+ 2074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:55:59.485315 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:55:59.485957 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:55:59.486872 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:59.486901 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:59.488566 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:59.488571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:59.490750 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:55:59.490775 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:55:59.491819 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:55:59.491826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:55:59.492134 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:55:59.496934 (+ 4800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:55:59.496971 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:55:59.499673 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:55:59.503955 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:55:59.503967 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:55:59.503989 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:55:59.504679 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:55:59.505220 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:55:59.505226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:55:59.505229 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:55:59.505528 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:55:59.506229 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:55:59.506238 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:55:59.506718 (+ 480us) client_negotiation.cc:769] Sending connection context
0929 20:55:59.507065 (+ 347us) client_negotiation.cc:240] Negotiation successful
0929 20:55:59.507735 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"thread_start_us":742,"threads_started":1}
I20260929 20:55:59.511564 13572 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37055
I20260929 20:55:59.511888 13572 heartbeater.cc:464] Registering TS with master...
I20260929 20:55:59.512763 13572 heartbeater.cc:511] Master 127.0.97.62:37055 requested a full tablet report, sending...
I20260929 20:55:59.515020 13231 ts_manager.cc:194] Registered new tserver with Master: bd796e8c39d74bd198de2e53f5dcb6ff (127.0.97.2:39483)
I20260929 20:55:59.517128 13231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47795
I20260929 20:55:59.800176 13428 heartbeater.cc:464] Registering TS with master...
I20260929 20:55:59.800647 13428 heartbeater.cc:511] Master 127.0.97.62:37055 requested a full tablet report, sending...
I20260929 20:55:59.801965 13231 ts_manager.cc:194] Registered new tserver with Master: 81d118f2752749a2b6497c8d5fe23a7e (127.0.97.1:44747)
I20260929 20:55:59.803174 13231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43589
Sep 29 20:55:59 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260929 20:55:59.904641 13576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:55:59.905143 13576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:55:59.905337 13576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:55:59.926468 13576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:55:59.926690 13576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:55:59.926925 13576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:55:59.942601 13576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:49039/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34275
--tserver_master_addrs=127.0.97.62:37055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:55:59.944540 13576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:55:59.946326 13576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:55:59.970486 13582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:55:59.970520 13583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:55:59.976570 13576 server_base.cc:1034] running on GCE node
W20260929 20:55:59.977299 13585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:55:59.978224 13576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:55:59.990563 13576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:55:59.991879 13576 hybrid_clock.cc:648] HybridClock initialized: now 1790715359991795 us; error 88 us; skew 500 ppm
I20260929 20:55:59.992516 13576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:55:59 dist-test-slave-m9g1 krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:55:59.997246 13576 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:56:00.042282 13576 webserver.cc:460] Webserver started at http://127.0.97.3:34275/ using document root <none> and password file <none>
I20260929 20:56:00.043542 13576 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:00.043774 13576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:56:00 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:56:00.094743 13576 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.006s sys 0.005s
I20260929 20:56:00.105083 13593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:00.107366 13576 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.010s sys 0.000s
I20260929 20:56:00.107652 13576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "7a3f0d6f70e1425b8a4d423ecc064801"
format_stamp: "Formatted at 2026-09-29 20:55:56 on dist-test-slave-m9g1"
server_key: "ead1641a28e6e2044cf23f33acca3658"
server_key_iv: "5ee11b3fdbe70b52f4057b2d1a82f428"
server_key_version: "kuduclusterkey@0"
I20260929 20:56:00.109278 13576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:00.147606 13576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:00.153570 13576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:00.153887 13576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:00.161423 13576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:56:00.169812 13576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:56:00.169942 13576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:00.170046 13576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:56:00.170112 13576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:56:00 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:56:00.434607 13576 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42893
I20260929 20:56:00.434671 13706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42893 every 8 connection(s)
I20260929 20:56:00.436790 13576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:56:00.444573 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 13576
Sep 29 20:56:00 dist-test-slave-m9g1 krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:56:00.487720 13710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:00.446589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.0.97.62:37055)
0929 20:56:00.467991 (+ 21402us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:00.467999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:00.468031 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:56:00.468126 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:00.468133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:00.468230 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:00.468379 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:00.472689 (+ 4310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.473949 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.475823 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.476484 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.476620 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:00.480029 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:00.480036 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:00.480041 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:00.480082 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:00.482929 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:00.483792 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:00.483811 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:00.483814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:00.483966 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:00.484411 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:00.484416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:00.484419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:00.484764 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:00.485027 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:00.487090 (+ 2063us) server_negotiation.cc:299] Negotiation successful
0929 20:56:00.487347 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21212,"thread_start_us":610,"threads_started":1}
I20260929 20:56:00.495033 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:00.443454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56175 (local address 127.0.97.62:37055)
0929 20:56:00.447116 (+ 3662us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:00.447123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:00.470437 (+ 23314us) server_negotiation.cc:407] Connection header received
0929 20:56:00.472601 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:00.472607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:00.472690 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:00.472800 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:00.479036 (+ 6236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.480221 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.482814 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.483426 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.483547 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:00.487793 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:00.487799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:00.487804 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:00.487850 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:00.491018 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:00.491967 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:00.491972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:00.491974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:00.492098 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:00.492668 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:00.492674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:00.492677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:00.493043 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:00.493193 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:00.494414 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0929 20:56:00.494638 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3532,"thread_start_us":25047,"threads_started":1}
I20260929 20:56:00.497975 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:00.442821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37055 (local address 127.0.97.3:56175)
0929 20:56:00.469336 (+ 26515us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:00.469415 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:00.472163 (+ 2748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:00.477122 (+ 4959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:00.477144 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:00.477845 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:00.478767 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:00.478797 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.480484 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.480490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:00.482622 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:00.482639 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.483851 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.483860 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:00.484171 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:00.484870 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:00.484907 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:00.487509 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:00.491179 (+ 3670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:00.491188 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:00.491207 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:00.491831 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:00.492203 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:00.492208 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:00.492212 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:00.492513 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:00.493265 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:00.493274 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:00.493941 (+ 667us) client_negotiation.cc:769] Sending connection context
0929 20:56:00.495576 (+ 1635us) client_negotiation.cc:240] Negotiation successful
0929 20:56:00.496262 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26050,"thread_start_us":31145,"threads_started":1}
I20260929 20:56:00.499957 13707 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37055
I20260929 20:56:00.500298 13707 heartbeater.cc:464] Registering TS with master...
I20260929 20:56:00.501185 13707 heartbeater.cc:511] Master 127.0.97.62:37055 requested a full tablet report, sending...
I20260929 20:56:00.503449 13230 ts_manager.cc:194] Registered new tserver with Master: 7a3f0d6f70e1425b8a4d423ecc064801 (127.0.97.3:42893)
I20260929 20:56:00.504830 13230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56175
I20260929 20:56:00.509317 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:56:00.522143 13572 heartbeater.cc:503] Master 127.0.97.62:37055 was elected leader, sending a full tablet report...
I20260929 20:56:00.538900 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:00.525264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59144 (local address 127.0.97.62:37055)
0929 20:56:00.525488 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:00.525494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:00.525831 (+ 337us) server_negotiation.cc:407] Connection header received
0929 20:56:00.526024 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:00.526027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:00.526089 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:00.526169 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:00.528075 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.528774 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.530787 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.531269 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.531367 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:00.533811 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:00.533816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:00.533819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:00.533851 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:00.536394 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:00.537086 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:00.537090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:00.537093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:00.537221 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:00.537603 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:00.537606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:00.537608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:00.537861 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:00.538050 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:00.538427 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 20:56:00.538602 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 20:56:00.543715 13230 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:56:00.551608 13230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:56:00.579087 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:00.568951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49366 (local address 127.0.97.1:44747)
0929 20:56:00.569621 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:00.569630 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:00.570228 (+ 598us) server_negotiation.cc:407] Connection header received
0929 20:56:00.570559 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:00.570567 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:00.570780 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:00.570958 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:00.571842 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.573374 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.576009 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.577486 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.577587 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:00.577832 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:00.578558 (+ 726us) server_negotiation.cc:299] Negotiation successful
0929 20:56:00.578739 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":312,"threads_started":1}
I20260929 20:56:00.579157 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:00.566502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42893 (local address 127.0.0.1:46052)
0929 20:56:00.568030 (+ 1528us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:00.568050 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:00.568248 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:00.569323 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:00.569330 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:00.569362 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:00.569783 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:00.569796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.574003 (+ 4207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.574008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:00.576113 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:00.576128 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.578403 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.578409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:00.578587 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:00.578618 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:56:00.578700 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 20:56:00.578826 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1416,"thread_start_us":820,"threads_started":1}
I20260929 20:56:00.579452 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:00.566516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46052 (local address 127.0.97.3:42893)
0929 20:56:00.567118 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:00.567127 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:00.568147 (+ 1020us) server_negotiation.cc:407] Connection header received
0929 20:56:00.568448 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:00.568457 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:00.568750 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:00.568936 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:00.570035 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.573812 (+ 3777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.576303 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.577822 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.577937 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:00.578153 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:00.578966 (+ 813us) server_negotiation.cc:299] Negotiation successful
0929 20:56:00.579139 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":289,"threads_started":1}
I20260929 20:56:00.580184 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:00.568525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44747 (local address 127.0.0.1:49366)
0929 20:56:00.570084 (+ 1559us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:00.570104 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:00.570350 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:00.571217 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:00.571221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:00.571247 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:00.571633 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:00.571643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.573562 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.573567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:00.575821 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:00.575834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.578055 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.578062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:00.578271 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:00.578333 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 20:56:00.578476 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 20:56:00.578692 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1436,"mutex_wait_us":22,"thread_start_us":2440,"threads_started":1}
I20260929 20:56:00.583150 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:00.566482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39483 (local address 127.0.0.1:36992)
0929 20:56:00.567735 (+ 1253us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:00.567782 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:00.567983 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:00.570091 (+ 2108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:00.570095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:00.570122 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:00.570498 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:00.570510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.572737 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.572742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:00.578084 (+ 5342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:00.578096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.581043 (+ 2947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.581049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:00.582531 (+ 1482us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:00.582557 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 20:56:00.582638 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 20:56:00.582779 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"mutex_wait_us":14,"thread_start_us":1619,"threads_started":1}
I20260929 20:56:00.583204 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:00.566754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36992 (local address 127.0.97.2:39483)
0929 20:56:00.568649 (+ 1895us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:00.568657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:00.569536 (+ 879us) server_negotiation.cc:407] Connection header received
0929 20:56:00.569645 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:00.569653 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:00.569896 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:00.570694 (+ 798us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:00.570925 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.572528 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.578364 (+ 5836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.580814 (+ 2450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.580943 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:00.581162 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:00.582716 (+ 1554us) server_negotiation.cc:299] Negotiation successful
0929 20:56:00.582893 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1740,"thread_start_us":1530,"threads_started":1}
I20260929 20:56:00.583554 13642 tablet_service.cc:1467] Processing CreateTablet for tablet ea841f5e7d4e44f68be58e0606c065e5 (DEFAULT_TABLE table=test-table [id=863da09017414c7ea5d09ce5d3171029]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:56:00.583926 13362 tablet_service.cc:1467] Processing CreateTablet for tablet ea841f5e7d4e44f68be58e0606c065e5 (DEFAULT_TABLE table=test-table [id=863da09017414c7ea5d09ce5d3171029]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:56:00.585438 13362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea841f5e7d4e44f68be58e0606c065e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:00.585554 13642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea841f5e7d4e44f68be58e0606c065e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:00.586959 13507 tablet_service.cc:1467] Processing CreateTablet for tablet ea841f5e7d4e44f68be58e0606c065e5 (DEFAULT_TABLE table=test-table [id=863da09017414c7ea5d09ce5d3171029]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:56:00.588455 13507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea841f5e7d4e44f68be58e0606c065e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:00.600879 13736 tablet_bootstrap.cc:492] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e: Bootstrap starting.
I20260929 20:56:00.603052 13735 tablet_bootstrap.cc:492] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801: Bootstrap starting.
I20260929 20:56:00.604493 13737 tablet_bootstrap.cc:492] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff: Bootstrap starting.
I20260929 20:56:00.606369 13735 tablet_bootstrap.cc:654] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:00.606463 13736 tablet_bootstrap.cc:654] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:00.607584 13735 log.cc:824] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:00.607782 13736 log.cc:824] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:00.607928 13737 tablet_bootstrap.cc:654] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:00.608966 13737 log.cc:824] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:00.610716 13735 tablet_bootstrap.cc:492] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801: No bootstrap required, opened a new log
I20260929 20:56:00.610978 13735 ts_tablet_manager.cc:1397] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260929 20:56:00.611701 13736 tablet_bootstrap.cc:492] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e: No bootstrap required, opened a new log
I20260929 20:56:00.612010 13736 ts_tablet_manager.cc:1397] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260929 20:56:00.612330 13737 tablet_bootstrap.cc:492] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff: No bootstrap required, opened a new log
I20260929 20:56:00.612617 13737 ts_tablet_manager.cc:1397] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 20:56:00.618538 13735 raft_consensus.cc:348] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3f0d6f70e1425b8a4d423ecc064801" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42893 } } peers { permanent_uuid: "bd796e8c39d74bd198de2e53f5dcb6ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39483 } } peers { permanent_uuid: "81d118f2752749a2b6497c8d5fe23a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44747 } }
I20260929 20:56:00.618920 13735 raft_consensus.cc:374] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:00.619035 13735 raft_consensus.cc:729] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a3f0d6f70e1425b8a4d423ecc064801, State: Initialized, Role: FOLLOWER
I20260929 20:56:00.619657 13735 consensus_queue.cc:260] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3f0d6f70e1425b8a4d423ecc064801" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42893 } } peers { permanent_uuid: "bd796e8c39d74bd198de2e53f5dcb6ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39483 } } peers { permanent_uuid: "81d118f2752749a2b6497c8d5fe23a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44747 } }
I20260929 20:56:00.620745 13707 heartbeater.cc:503] Master 127.0.97.62:37055 was elected leader, sending a full tablet report...
I20260929 20:56:00.621477 13735 ts_tablet_manager.cc:1428] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260929 20:56:00.622457 13736 raft_consensus.cc:348] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3f0d6f70e1425b8a4d423ecc064801" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42893 } } peers { permanent_uuid: "bd796e8c39d74bd198de2e53f5dcb6ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39483 } } peers { permanent_uuid: "81d118f2752749a2b6497c8d5fe23a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44747 } }
I20260929 20:56:00.623003 13736 raft_consensus.cc:374] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:00.623135 13736 raft_consensus.cc:729] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d118f2752749a2b6497c8d5fe23a7e, State: Initialized, Role: FOLLOWER
I20260929 20:56:00.622924 13737 raft_consensus.cc:348] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3f0d6f70e1425b8a4d423ecc064801" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42893 } } peers { permanent_uuid: "bd796e8c39d74bd198de2e53f5dcb6ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39483 } } peers { permanent_uuid: "81d118f2752749a2b6497c8d5fe23a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44747 } }
I20260929 20:56:00.623411 13737 raft_consensus.cc:374] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:00.623543 13737 raft_consensus.cc:729] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd796e8c39d74bd198de2e53f5dcb6ff, State: Initialized, Role: FOLLOWER
I20260929 20:56:00.623932 13736 consensus_queue.cc:260] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3f0d6f70e1425b8a4d423ecc064801" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42893 } } peers { permanent_uuid: "bd796e8c39d74bd198de2e53f5dcb6ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39483 } } peers { permanent_uuid: "81d118f2752749a2b6497c8d5fe23a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44747 } }
I20260929 20:56:00.624115 13737 consensus_queue.cc:260] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3f0d6f70e1425b8a4d423ecc064801" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42893 } } peers { permanent_uuid: "bd796e8c39d74bd198de2e53f5dcb6ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39483 } } peers { permanent_uuid: "81d118f2752749a2b6497c8d5fe23a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44747 } }
I20260929 20:56:00.625058 13428 heartbeater.cc:503] Master 127.0.97.62:37055 was elected leader, sending a full tablet report...
I20260929 20:56:00.625799 13736 ts_tablet_manager.cc:1428] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260929 20:56:00.627578 13737 ts_tablet_manager.cc:1428] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260929 20:56:00.725804 13708 tablet.cc:2369] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:56:00.728294 13742 raft_consensus.cc:482] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:56:00.728544 13742 raft_consensus.cc:504] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3f0d6f70e1425b8a4d423ecc064801" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42893 } } peers { permanent_uuid: "bd796e8c39d74bd198de2e53f5dcb6ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39483 } } peers { permanent_uuid: "81d118f2752749a2b6497c8d5fe23a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44747 } }
I20260929 20:56:00.730413 13742 leader_election.cc:288] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a3f0d6f70e1425b8a4d423ecc064801 (127.0.97.3:42893), bd796e8c39d74bd198de2e53f5dcb6ff (127.0.97.2:39483)
I20260929 20:56:00.736922 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:00.730942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59615 (local address 127.0.97.2:39483)
0929 20:56:00.731186 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:00.731192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:00.731657 (+ 465us) server_negotiation.cc:407] Connection header received
0929 20:56:00.731819 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:00.731824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:00.731899 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:00.731963 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:00.732562 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.733299 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.735227 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.736006 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.736126 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:00.736222 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:00.736562 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 20:56:00.736684 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 20:56:00.736940 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:00.730779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38065 (local address 127.0.97.3:42893)
0929 20:56:00.731006 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:00.731013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:00.731742 (+ 729us) server_negotiation.cc:407] Connection header received
0929 20:56:00.731902 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:00.731907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:00.731973 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:00.732057 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:00.732631 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.733319 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.735234 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.736010 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.736120 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:00.736224 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:00.736537 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 20:56:00.736662 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 20:56:00.737084 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:00.730772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42893 (local address 127.0.97.1:38065)
0929 20:56:00.731625 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:00.731642 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:00.731769 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:00.732090 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:00.732094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:00.732126 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:00.732484 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:00.732494 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.733440 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.733443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:00.735096 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:00.735108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.736224 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.736231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:00.736366 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:00.736420 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 20:56:00.736494 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 20:56:00.736619 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":55,"thread_start_us":573,"threads_started":1}
I20260929 20:56:00.737228 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:00.730710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39483 (local address 127.0.97.1:59615)
0929 20:56:00.731517 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:00.731538 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:00.731683 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:00.731999 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:00.732004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:00.732027 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:00.732414 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:00.732424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.733467 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.733471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:00.735083 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:00.735095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.736267 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.736272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:00.736417 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:00.736439 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 20:56:00.736520 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 20:56:00.736642 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":57,"spinlock_wait_cycles":3968,"thread_start_us":435,"threads_started":1}
W20260929 20:56:00.737380 13429 tablet.cc:2369] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:56:00.738294 13527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea841f5e7d4e44f68be58e0606c065e5" candidate_uuid: "81d118f2752749a2b6497c8d5fe23a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd796e8c39d74bd198de2e53f5dcb6ff" is_pre_election: true
I20260929 20:56:00.738356 13662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea841f5e7d4e44f68be58e0606c065e5" candidate_uuid: "81d118f2752749a2b6497c8d5fe23a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a3f0d6f70e1425b8a4d423ecc064801" is_pre_election: true
W20260929 20:56:00.738726 13573 tablet.cc:2369] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:56:00.738816 13527 raft_consensus.cc:2454] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81d118f2752749a2b6497c8d5fe23a7e in term 0.
I20260929 20:56:00.738886 13662 raft_consensus.cc:2454] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81d118f2752749a2b6497c8d5fe23a7e in term 0.
I20260929 20:56:00.739449 13315 leader_election.cc:302] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81d118f2752749a2b6497c8d5fe23a7e, bd796e8c39d74bd198de2e53f5dcb6ff; no voters:
I20260929 20:56:00.739818 13742 raft_consensus.cc:2781] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:56:00.739948 13742 raft_consensus.cc:482] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:56:00.740052 13742 raft_consensus.cc:3037] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:00.742239 13742 raft_consensus.cc:504] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3f0d6f70e1425b8a4d423ecc064801" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42893 } } peers { permanent_uuid: "bd796e8c39d74bd198de2e53f5dcb6ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39483 } } peers { permanent_uuid: "81d118f2752749a2b6497c8d5fe23a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44747 } }
I20260929 20:56:00.742815 13742 leader_election.cc:288] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [CANDIDATE]: Term 1 election: Requested vote from peers 7a3f0d6f70e1425b8a4d423ecc064801 (127.0.97.3:42893), bd796e8c39d74bd198de2e53f5dcb6ff (127.0.97.2:39483)
I20260929 20:56:00.743330 13662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea841f5e7d4e44f68be58e0606c065e5" candidate_uuid: "81d118f2752749a2b6497c8d5fe23a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a3f0d6f70e1425b8a4d423ecc064801"
I20260929 20:56:00.743327 13527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea841f5e7d4e44f68be58e0606c065e5" candidate_uuid: "81d118f2752749a2b6497c8d5fe23a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd796e8c39d74bd198de2e53f5dcb6ff"
I20260929 20:56:00.743510 13662 raft_consensus.cc:3037] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:00.743549 13527 raft_consensus.cc:3037] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:00.745616 13662 raft_consensus.cc:2454] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81d118f2752749a2b6497c8d5fe23a7e in term 1.
I20260929 20:56:00.745626 13527 raft_consensus.cc:2454] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81d118f2752749a2b6497c8d5fe23a7e in term 1.
I20260929 20:56:00.746011 13316 leader_election.cc:302] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a3f0d6f70e1425b8a4d423ecc064801, 81d118f2752749a2b6497c8d5fe23a7e; no voters:
I20260929 20:56:00.746312 13742 raft_consensus.cc:2781] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:56:00.746769 13742 raft_consensus.cc:686] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [term 1 LEADER]: Becoming Leader. State: Replica: 81d118f2752749a2b6497c8d5fe23a7e, State: Running, Role: LEADER
I20260929 20:56:00.747225 13742 consensus_queue.cc:237] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3f0d6f70e1425b8a4d423ecc064801" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42893 } } peers { permanent_uuid: "bd796e8c39d74bd198de2e53f5dcb6ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39483 } } peers { permanent_uuid: "81d118f2752749a2b6497c8d5fe23a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44747 } }
I20260929 20:56:00.751399 13230 catalog_manager.cc:5495] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 81d118f2752749a2b6497c8d5fe23a7e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "81d118f2752749a2b6497c8d5fe23a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3f0d6f70e1425b8a4d423ecc064801" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd796e8c39d74bd198de2e53f5dcb6ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d118f2752749a2b6497c8d5fe23a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44747 } health_report { overall_health: HEALTHY } } }
I20260929 20:56:00.775894 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:00.770612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49368 (local address 127.0.97.1:44747)
0929 20:56:00.770830 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:00.770837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:00.770858 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:56:00.770955 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:00.770959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:00.771047 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:00.771177 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:56:00.771581 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.772326 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.773701 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.774095 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.774237 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:00.774785 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:56:00.775298 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:56:00.775458 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:00.775534 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 20:56:00.775621 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:56:00.784169 13527 raft_consensus.cc:1261] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff [term 1 FOLLOWER]: Refusing update from remote peer 81d118f2752749a2b6497c8d5fe23a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:56:00.784169 13662 raft_consensus.cc:1261] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801 [term 1 FOLLOWER]: Refusing update from remote peer 81d118f2752749a2b6497c8d5fe23a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:56:00.785216 13746 consensus_queue.cc:1035] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd796e8c39d74bd198de2e53f5dcb6ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:56:00.785550 13742 consensus_queue.cc:1035] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a3f0d6f70e1425b8a4d423ecc064801" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:56:00.796406 13749 mvcc.cc:205] Tried to move back new op lower bound from 7334770117761302528 to 7334770117624029184. Current Snapshot: MvccSnapshot[applied={T|T < 7334770117761302528}]
I20260929 20:56:00.822062 13230 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:56:00.822396 13230 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:56:00.826241 13230 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49a2a96c909e445dad819a9126e03162: Sending DeleteTablet for 3 replicas of tablet ea841f5e7d4e44f68be58e0606c065e5
I20260929 20:56:00.827435 13642 tablet_service.cc:1514] Processing DeleteTablet for tablet ea841f5e7d4e44f68be58e0606c065e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:56:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46052
I20260929 20:56:00.827477 13507 tablet_service.cc:1514] Processing DeleteTablet for tablet ea841f5e7d4e44f68be58e0606c065e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:56:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36992
I20260929 20:56:00.827625 13362 tablet_service.cc:1514] Processing DeleteTablet for tablet ea841f5e7d4e44f68be58e0606c065e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:56:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49366
I20260929 20:56:00.831007 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 13268
I20260929 20:56:00.839980 13759 tablet_replica.cc:330] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801: stopping tablet replica
I20260929 20:56:00.840705 13759 raft_consensus.cc:2229] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:56:00.841670 13759 raft_consensus.cc:2258] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:56:00.842612 13760 tablet_replica.cc:330] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff: stopping tablet replica
I20260929 20:56:00.843144 13760 raft_consensus.cc:2229] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:56:00.843683 13760 raft_consensus.cc:2258] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:56:00.845399 13761 tablet_replica.cc:330] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e: stopping tablet replica
I20260929 20:56:00.845877 13761 raft_consensus.cc:2229] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:56:00.846009 13759 ts_tablet_manager.cc:1905] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:56:00.846378 13760 ts_tablet_manager.cc:1905] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:56:00.846596 13761 raft_consensus.cc:2258] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:56:00.852159 13761 ts_tablet_manager.cc:1905] T ea841f5e7d4e44f68be58e0606c065e5 P 81d118f2752749a2b6497c8d5fe23a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:56:00.853170 13759 ts_tablet_manager.cc:1918] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:56:00.853303 13760 ts_tablet_manager.cc:1918] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:56:00.853451 13759 log.cc:1196] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/ea841f5e7d4e44f68be58e0606c065e5
I20260929 20:56:00.853483 13760 log.cc:1196] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/ea841f5e7d4e44f68be58e0606c065e5
I20260929 20:56:00.854035 13760 ts_tablet_manager.cc:1939] T ea841f5e7d4e44f68be58e0606c065e5 P bd796e8c39d74bd198de2e53f5dcb6ff: Deleting consensus metadata
I20260929 20:56:00.854043 13759 ts_tablet_manager.cc:1939] T ea841f5e7d4e44f68be58e0606c065e5 P 7a3f0d6f70e1425b8a4d423ecc064801: Deleting consensus metadata
I20260929 20:56:00.855904 13215 catalog_manager.cc:4841] TS bd796e8c39d74bd198de2e53f5dcb6ff (127.0.97.2:39483): tablet ea841f5e7d4e44f68be58e0606c065e5 (table test-table [id=863da09017414c7ea5d09ce5d3171029]) successfully deleted
I20260929 20:56:00.855933 13216 catalog_manager.cc:4841] TS 7a3f0d6f70e1425b8a4d423ecc064801 (127.0.97.3:42893): tablet ea841f5e7d4e44f68be58e0606c065e5 (table test-table [id=863da09017414c7ea5d09ce5d3171029]) successfully deleted
I20260929 20:56:00.857365 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:00.832837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49372 (local address 127.0.97.1:44747)
0929 20:56:00.833303 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:00.833308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:00.833326 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:56:00.833385 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:00.833391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:00.833487 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:00.833586 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:00.836574 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.837842 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.839914 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.840610 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.840740 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:00.844579 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:00.844595 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:00.844612 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:00.844654 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:00.849444 (+ 4790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:00.852614 (+ 3170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:00.852626 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:00.852629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:00.852822 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:00.854635 (+ 1813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:00.854640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:00.854644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:00.855120 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:00.855421 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:00.856732 (+ 1311us) server_negotiation.cc:299] Negotiation successful
0929 20:56:00.857001 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350}
I20260929 20:56:00.858273 13423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:00.954813 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 13268
W20260929 20:56:00.968479 13215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44747 (error 108)
I20260929 20:56:00.969620 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 13432
I20260929 20:56:00.970216 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:00.969753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44747 (local address 127.0.0.1:49376)
0929 20:56:00.969915 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:00.970019 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 20:56:00.970533 13215 catalog_manager.cc:4584] TS 81d118f2752749a2b6497c8d5fe23a7e (127.0.97.1:44747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea841f5e7d4e44f68be58e0606c065e5: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44747: connect: Connection refused (error 111)
I20260929 20:56:00.983011 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:00.970531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37000 (local address 127.0.97.2:39483)
0929 20:56:00.970689 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:00.970693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:00.970713 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:56:00.970883 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:00.970887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:00.970955 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:00.971042 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:00.972298 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.973054 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.974291 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:00.974759 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:00.974858 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:00.977265 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:00.977281 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:00.977298 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:00.977336 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:00.980307 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:00.980988 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:00.980995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:00.980997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:00.981138 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:00.981521 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:00.981525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:00.981527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:00.981863 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:00.982100 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:00.982501 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 20:56:00.982673 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:56:00.983713 13567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:01.024274 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:01.023808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44747 (local address 127.0.0.1:49388)
0929 20:56:01.024026 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:01.024131 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 20:56:01.025503 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:01.025069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44747 (local address 127.0.0.1:49400)
0929 20:56:01.025238 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:01.025336 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 20:56:01.085695 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 13432
I20260929 20:56:01.086062 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:01.085571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44747 (local address 127.0.0.1:49416)
0929 20:56:01.085818 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:01.085907 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260929 20:56:01.088233 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:01.087808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44747 (local address 127.0.0.1:49424)
0929 20:56:01.087972 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:01.088079 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 20:56:01.099382 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 13576
I20260929 20:56:01.112438 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:01.100151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46066 (local address 127.0.97.3:42893)
0929 20:56:01.100338 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:01.100343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:01.100362 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:56:01.100554 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:01.100559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:01.100622 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:01.100702 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:01.102038 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:01.102852 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:01.104133 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:01.104555 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:01.104654 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:01.107245 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:01.107261 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:01.107277 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:01.107317 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:01.110034 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:01.110716 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:01.110723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:01.110726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:01.110863 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:01.111145 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:01.111148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:01.111149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:01.111549 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:01.111771 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:01.111976 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 20:56:01.112136 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:56:01.113137 13702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:01.169423 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:01.168903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44747 (local address 127.0.0.1:49434)
0929 20:56:01.169157 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:01.169250 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260929 20:56:01.170547 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:01.170129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44747 (local address 127.0.0.1:49450)
0929 20:56:01.170343 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:01.170427 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260929 20:56:01.201145 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 13576
I20260929 20:56:01.214648 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 13198
I20260929 20:56:01.215833 13261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:01.327304 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 13198
I20260929 20:56:01.348784 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 20:56:01.684135 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 20:56:01.684347 388 mini_ranger.cc:66] Stopping Ranger...
I20260929 20:56:02.019969 388 mini_ranger.cc:68] Stopped Ranger
2026-09-29 20:56:02.020 UTC [12250] LOG: received smart shutdown request
2026-09-29 20:56:02.023 UTC [12250] LOG: background worker "logical replication launcher" (PID 12259) exited with exit code 1
2026-09-29 20:56:02.023 UTC [12254] LOG: shutting down
2026-09-29 20:56:02.204 UTC [12250] LOG: database system is shut down
2026-09-29T20:56:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52537 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:56:02 dist-test-slave-m9g1 krb5kdc[13775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:56:02 dist-test-slave-m9g1 krb5kdc[13775](info): set up 2 sockets
Sep 29 20:56:02 dist-test-slave-m9g1 krb5kdc[13775](info): commencing operation
krb5kdc: starting...
W20260929 20:56:03.326273 388 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:56:03 dist-test-slave-m9g1 krb5kdc[13775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:56:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:56:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:56:03.543845 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39953
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39953,127.0.97.61:41147,127.0.97.60:40373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:56:03.873016 13788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:03.873315 13788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:03.873461 13788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:03.880825 13788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:56:03.880956 13788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:03.881012 13788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:03.881093 13788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:56:03.881155 13788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:56:03.891364 13788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39953,127.0.97.61:41147,127.0.97.60:40373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:03.892726 13788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:03.894101 13788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:03.906315 13793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:03.906389 13794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:03.906579 13796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:03.907142 13788 server_base.cc:1034] running on GCE node
I20260929 20:56:03.908025 13788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:03.909477 13788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:03.910683 13788 hybrid_clock.cc:648] HybridClock initialized: now 1790715363910628 us; error 48 us; skew 500 ppm
I20260929 20:56:03.911052 13788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:03 dist-test-slave-m9g1 krb5kdc[13775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:03.914165 13788 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:56:03.915705 13788 webserver.cc:460] Webserver started at http://127.0.97.62:34635/ using document root <none> and password file <none>
I20260929 20:56:03.916399 13788 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:03.916495 13788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:03.916764 13788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:56:03.919857 13788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "9216a20083ef4edaa0190753b702dfe3"
format_stamp: "Formatted at 2026-09-29 20:56:03 on dist-test-slave-m9g1"
server_key: "c3c8d4ec521c9267c7656f6e513d8249"
server_key_iv: "6654abe41fa023bf4a9ac08b61dcb9f8"
server_key_version: "encryptionkey@0"
I20260929 20:56:03.920607 13788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "9216a20083ef4edaa0190753b702dfe3"
format_stamp: "Formatted at 2026-09-29 20:56:03 on dist-test-slave-m9g1"
server_key: "c3c8d4ec521c9267c7656f6e513d8249"
server_key_iv: "6654abe41fa023bf4a9ac08b61dcb9f8"
server_key_version: "encryptionkey@0"
I20260929 20:56:03.925805 13788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:56:03.929327 13803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:03.930783 13788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:56:03.930941 13788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "9216a20083ef4edaa0190753b702dfe3"
format_stamp: "Formatted at 2026-09-29 20:56:03 on dist-test-slave-m9g1"
server_key: "c3c8d4ec521c9267c7656f6e513d8249"
server_key_iv: "6654abe41fa023bf4a9ac08b61dcb9f8"
server_key_version: "encryptionkey@0"
I20260929 20:56:03.931125 13788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:03.956264 13788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:03.960465 13788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:03.960738 13788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:03.978121 13788 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39953
I20260929 20:56:03.978147 13854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39953 every 8 connection(s)
I20260929 20:56:03.979578 13788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:56:03.983572 13855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:03.985083 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 13788
I20260929 20:56:03.985275 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:56:03.985924 388 external_mini_cluster.cc:1428] Setting key e9e2fec67836b84ded4f45447b17a863
I20260929 20:56:03.990491 13855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 29 20:56:04 dist-test-slave-m9g1 krb5kdc[13775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:56:04.013968 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:03.993825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39953 (local address 127.0.0.1:53476)
0929 20:56:03.994680 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:03.994743 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:03.996722 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:03.997574 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:03.997581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:03.998386 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:03.998729 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:03.998758 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.000347 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.000352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:04.001443 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:04.001456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.002369 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.002376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:04.002533 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:04.003147 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:04.003180 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:04.006638 (+ 3458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:04.009771 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:04.009783 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:04.009797 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:04.010214 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:04.010611 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:04.010616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:04.010620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:04.010785 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:04.011560 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:04.011567 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:04.011791 (+ 224us) client_negotiation.cc:769] Sending connection context
0929 20:56:04.012021 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 20:56:04.012622 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":135,"thread_start_us":384,"threads_started":1}
I20260929 20:56:04.013936 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:03.993826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.0.97.62:39953)
0929 20:56:03.995542 (+ 1716us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:03.995551 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:03.995603 (+ 52us) server_negotiation.cc:407] Connection header received
0929 20:56:03.997102 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:03.997107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:03.997359 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:03.997534 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:03.998928 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.000204 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.001577 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.002123 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.002328 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:04.006858 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:04.006868 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:04.006880 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:04.006912 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:04.009640 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:04.010342 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:04.010349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:04.010351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:04.010506 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:04.010896 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:04.010899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:04.010901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:04.011430 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:04.011617 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:04.012175 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 20:56:04.012469 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1509,"spinlock_wait_cycles":24960,"thread_start_us":1291,"threads_started":1}
I20260929 20:56:04.017175 13855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:56:04.018326 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.017901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41147 (local address 127.0.0.1:59984)
0929 20:56:04.018065 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.018172 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 20:56:04.018584 13806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41147: connect: Connection refused (error 111)
I20260929 20:56:04.020078 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.019735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41147 (local address 127.0.0.1:59996)
0929 20:56:04.019879 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.019956 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 20:56:04.020272 13855 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41147: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41147: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 20:56:04.031558 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41147
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39953,127.0.97.61:41147,127.0.97.60:40373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:56:04.069619 13855 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } attempt: 1
I20260929 20:56:04.070894 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.070327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41147 (local address 127.0.0.1:60002)
0929 20:56:04.070569 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.070690 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 20:56:04.071990 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.071586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41147 (local address 127.0.0.1:60006)
0929 20:56:04.071740 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.071831 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 20:56:04.072193 13855 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41147: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41147: connect: Connection refused (error 111)
I20260929 20:56:04.140462 13855 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } attempt: 2
I20260929 20:56:04.141755 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.141164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41147 (local address 127.0.0.1:60018)
0929 20:56:04.141441 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.141573 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 20:56:04.142884 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.142486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41147 (local address 127.0.0.1:60034)
0929 20:56:04.142630 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.142730 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 20:56:04.143076 13855 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41147: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41147: connect: Connection refused (error 111)
I20260929 20:56:04.234350 13855 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } attempt: 3
I20260929 20:56:04.235687 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.235111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41147 (local address 127.0.0.1:60044)
0929 20:56:04.235355 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.235487 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 20:56:04.236862 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.236378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41147 (local address 127.0.0.1:60054)
0929 20:56:04.236548 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.236647 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 20:56:04.237067 13855 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41147: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41147: connect: Connection refused (error 111)
W20260929 20:56:04.355595 13862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:04.355892 13862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:04.355991 13862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:04.363142 13862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:56:04.363257 13862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:04.363301 13862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:04.363356 13862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:56:04.363399 13862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:56:04.373354 13862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39953,127.0.97.61:41147,127.0.97.60:40373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:04.374679 13862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:04.375933 13862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 20:56:04.380363 13855 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } attempt: 4
I20260929 20:56:04.381839 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.381118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41147 (local address 127.0.0.1:60064)
0929 20:56:04.381538 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.381658 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261}
I20260929 20:56:04.383102 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.382764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41147 (local address 127.0.0.1:60066)
0929 20:56:04.382916 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.382985 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 20:56:04.383313 13855 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41147: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41147: connect: Connection refused (error 111)
W20260929 20:56:04.387779 13867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:04.387749 13870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:04.387694 13868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:04.388574 13862 server_base.cc:1034] running on GCE node
I20260929 20:56:04.389343 13862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:04.390878 13862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:04.392107 13862 hybrid_clock.cc:648] HybridClock initialized: now 1790715364392052 us; error 48 us; skew 500 ppm
I20260929 20:56:04.392457 13862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:04 dist-test-slave-m9g1 krb5kdc[13775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:04.395592 13862 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 20:56:04.397140 13862 webserver.cc:460] Webserver started at http://127.0.97.61:33435/ using document root <none> and password file <none>
I20260929 20:56:04.397878 13862 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:04.397998 13862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:04.398334 13862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:56:04.401473 13862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-1/data/instance:
uuid: "ea685a1b25f14d918281710e627a686a"
format_stamp: "Formatted at 2026-09-29 20:56:04 on dist-test-slave-m9g1"
server_key: "aa6823e859cde3aed33ed05e6cea9911"
server_key_iv: "a73ccabd80d18eb05dd0c4a59e729014"
server_key_version: "encryptionkey@0"
I20260929 20:56:04.402379 13862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-1/wal/instance:
uuid: "ea685a1b25f14d918281710e627a686a"
format_stamp: "Formatted at 2026-09-29 20:56:04 on dist-test-slave-m9g1"
server_key: "aa6823e859cde3aed33ed05e6cea9911"
server_key_iv: "a73ccabd80d18eb05dd0c4a59e729014"
server_key_version: "encryptionkey@0"
I20260929 20:56:04.407742 13862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 20:56:04.411381 13877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:04.412856 13862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:56:04.413023 13862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-1/wal
uuid: "ea685a1b25f14d918281710e627a686a"
format_stamp: "Formatted at 2026-09-29 20:56:04 on dist-test-slave-m9g1"
server_key: "aa6823e859cde3aed33ed05e6cea9911"
server_key_iv: "a73ccabd80d18eb05dd0c4a59e729014"
server_key_version: "encryptionkey@0"
I20260929 20:56:04.413213 13862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:04.430634 13862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:04.435775 13862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:04.436008 13862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:04.452093 13862 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:41147
I20260929 20:56:04.452116 13928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:41147 every 8 connection(s)
I20260929 20:56:04.453462 13862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-1/data/info.pb
I20260929 20:56:04.457239 13929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:04.462826 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 13862
I20260929 20:56:04.463032 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-1/wal/instance
I20260929 20:56:04.463543 388 external_mini_cluster.cc:1428] Setting key 804209c273e7c984f914fa7446c0b33b
I20260929 20:56:04.464140 13929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:56:04 dist-test-slave-m9g1 krb5kdc[13775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260929 20:56:04.486563 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.467426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53490 (local address 127.0.97.62:39953)
0929 20:56:04.467631 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:04.467638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:04.468910 (+ 1272us) server_negotiation.cc:407] Connection header received
0929 20:56:04.471422 (+ 2512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:04.471429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:04.471523 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:04.471627 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:04.473148 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.474149 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.475736 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.476366 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.476492 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:04.479932 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:04.479940 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:04.479945 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:04.479993 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:04.483518 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:04.484304 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:04.484311 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:04.484315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:04.484470 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:04.484861 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:04.484865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:04.484867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:04.485194 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:04.485337 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:04.486050 (+ 713us) server_negotiation.cc:299] Negotiation successful
0929 20:56:04.486291 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:56:04.487932 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.467895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39953 (local address 127.0.0.1:53490)
0929 20:56:04.468707 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.468763 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:04.471048 (+ 2285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:04.471827 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:04.471835 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:04.472356 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:04.472956 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:04.472975 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.474320 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.474323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:04.475593 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:04.475610 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.476564 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.476568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:04.476750 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:04.477412 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:04.477436 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:04.479638 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:04.483679 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:04.483687 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:04.483703 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:04.484172 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:04.484564 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:04.484567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:04.484569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:04.484755 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:04.485309 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:04.485315 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:04.485668 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 20:56:04.485973 (+ 305us) client_negotiation.cc:240] Negotiation successful
0929 20:56:04.486497 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":316,"threads_started":1}
I20260929 20:56:04.489760 13929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:56:04 dist-test-slave-m9g1 krb5kdc[13775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 20:56:04.514714 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.490533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41147 (local address 127.0.0.1:60068)
0929 20:56:04.490746 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.490767 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:04.490877 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:04.491856 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:04.491860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:04.492297 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:04.492475 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:04.492485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.494101 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.494105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:04.495199 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:04.495212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.496041 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.496046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:04.496233 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:04.504140 (+ 7907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:04.504169 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:04.508486 (+ 4317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:04.511939 (+ 3453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:04.511943 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:04.511946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:04.512442 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:04.512809 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:04.512813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:04.512816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:04.512961 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:04.513709 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:04.513714 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:04.513853 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 20:56:04.513974 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 20:56:04.514382 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 20:56:04.514874 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.490778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60068 (local address 127.0.97.61:41147)
0929 20:56:04.491428 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:04.491434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:04.491459 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:56:04.491503 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:04.491509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:04.491696 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:04.491817 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:04.492638 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.493951 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.495337 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.495902 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.495996 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:04.508792 (+ 12796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:04.508812 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:04.508835 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:04.508875 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:04.511771 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:04.512567 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:04.512575 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:04.512577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:04.512704 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:04.513102 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:04.513107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:04.513110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:04.513544 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:04.513774 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:04.514135 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 20:56:04.514536 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":490,"thread_start_us":359,"threads_started":1}
I20260929 20:56:04.515477 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40373
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39953,127.0.97.61:41147,127.0.97.60:40373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:56:04.517586 13929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:56:04.518748 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.518250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49220)
0929 20:56:04.518422 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.518573 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 20:56:04.519047 13879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
I20260929 20:56:04.520562 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.520195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49222)
0929 20:56:04.520340 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.520438 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 20:56:04.520743 13929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40373: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
I20260929 20:56:04.570101 13929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } attempt: 1
I20260929 20:56:04.571552 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.570906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49234)
0929 20:56:04.571209 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.571335 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 20:56:04.572808 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.572365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49248)
0929 20:56:04.572549 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.572658 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 20:56:04.573159 13929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40373: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
I20260929 20:56:04.641376 13929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } attempt: 2
I20260929 20:56:04.642597 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.642110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49256)
0929 20:56:04.642333 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.642446 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 20:56:04.643638 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.643301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49268)
0929 20:56:04.643449 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.643521 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 20:56:04.643858 13929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40373: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
I20260929 20:56:04.682578 13855 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } attempt: 5
Sep 29 20:56:04 dist-test-slave-m9g1 krb5kdc[13775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:56:04.696861 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.683382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41147 (local address 127.0.0.1:60078)
0929 20:56:04.683618 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.683641 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:04.683763 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:04.684189 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:04.684193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:04.684579 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:04.684755 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:04.684766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.686269 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.686274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:04.687197 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:04.687212 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.688138 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.688142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:04.688289 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:04.688917 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:04.688935 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:04.691104 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:04.694693 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:04.694697 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:04.694700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:04.695061 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:04.695463 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:04.695466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:04.695469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:04.695585 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:04.696179 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:04.696182 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:04.696300 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 20:56:04.696414 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 20:56:04.696620 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 20:56:04.697148 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.683570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60078 (local address 127.0.97.61:41147)
0929 20:56:04.683776 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:04.683784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:04.683844 (+ 60us) server_negotiation.cc:407] Connection header received
0929 20:56:04.683918 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:04.683923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:04.684031 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:04.684141 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:04.684935 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.686049 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.687344 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.688001 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.688134 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:04.691326 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:04.691333 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:04.691338 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:04.691379 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:04.694573 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:04.695179 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:04.695185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:04.695189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:04.695364 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:04.695677 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:04.695682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:04.695686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:04.696078 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:04.696239 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:04.696564 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 20:56:04.696813 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:56:04.698153 13855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:56:04.699417 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.698950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49274)
0929 20:56:04.699139 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.699235 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 20:56:04.700409 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.700039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49286)
0929 20:56:04.700173 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.700280 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 20:56:04.700606 13855 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40373: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
I20260929 20:56:04.735055 13929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } attempt: 3
I20260929 20:56:04.736176 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.735688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49294)
0929 20:56:04.735906 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.736010 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 20:56:04.737317 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.736889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49298)
0929 20:56:04.737121 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.737207 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260929 20:56:04.737584 13929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40373: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
I20260929 20:56:04.751824 13855 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } attempt: 1
I20260929 20:56:04.753072 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.752551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49304)
0929 20:56:04.752745 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.752855 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 20:56:04.754068 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.753763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49306)
0929 20:56:04.753889 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.753962 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 20:56:04.754323 13855 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40373: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
I20260929 20:56:04.822556 13855 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } attempt: 2
I20260929 20:56:04.823693 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.823237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49314)
0929 20:56:04.823445 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.823534 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 20:56:04.824644 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.824352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49322)
0929 20:56:04.824474 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.824542 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 20:56:04.824849 13855 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40373: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
W20260929 20:56:04.841506 13936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:04.841782 13936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:04.841907 13936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:04.849191 13936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:56:04.849296 13936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:04.849367 13936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:04.849447 13936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:56:04.849507 13936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:56:04.859447 13936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39953,127.0.97.61:41147,127.0.97.60:40373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:04.860782 13936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:04.862092 13936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:04.873764 13942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:04.873780 13941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:04.873708 13944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:04.874392 13936 server_base.cc:1034] running on GCE node
I20260929 20:56:04.875159 13936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:04.876544 13936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:04.877739 13936 hybrid_clock.cc:648] HybridClock initialized: now 1790715364877689 us; error 48 us; skew 500 ppm
I20260929 20:56:04.878073 13936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:04 dist-test-slave-m9g1 krb5kdc[13775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:04.880815 13929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } attempt: 4
I20260929 20:56:04.881105 13936 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 20:56:04.881966 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.881396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49336)
0929 20:56:04.881673 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.881781 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 20:56:04.882902 13936 webserver.cc:460] Webserver started at http://127.0.97.60:41759/ using document root <none> and password file <none>
I20260929 20:56:04.882951 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.882630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49344)
0929 20:56:04.882770 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.882838 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 20:56:04.883162 13929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40373: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
I20260929 20:56:04.883635 13936 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:04.883750 13936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:04.884039 13936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:56:04.887089 13936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-2/data/instance:
uuid: "d8d15958932b423c956bf7c19c256e0f"
format_stamp: "Formatted at 2026-09-29 20:56:04 on dist-test-slave-m9g1"
server_key: "395078f07c4cb7c35d8555f532951410"
server_key_iv: "2454beb3b081f9533be22f0441f2e4b1"
server_key_version: "encryptionkey@0"
I20260929 20:56:04.887898 13936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-2/wal/instance:
uuid: "d8d15958932b423c956bf7c19c256e0f"
format_stamp: "Formatted at 2026-09-29 20:56:04 on dist-test-slave-m9g1"
server_key: "395078f07c4cb7c35d8555f532951410"
server_key_iv: "2454beb3b081f9533be22f0441f2e4b1"
server_key_version: "encryptionkey@0"
I20260929 20:56:04.893090 13936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:56:04.896487 13951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:04.897914 13936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:56:04.898075 13936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-2/wal
uuid: "d8d15958932b423c956bf7c19c256e0f"
format_stamp: "Formatted at 2026-09-29 20:56:04 on dist-test-slave-m9g1"
server_key: "395078f07c4cb7c35d8555f532951410"
server_key_iv: "2454beb3b081f9533be22f0441f2e4b1"
server_key_version: "encryptionkey@0"
I20260929 20:56:04.898299 13936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:04.917671 13936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:04.921676 13936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:04.921947 13936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:04.931046 13855 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } attempt: 3
I20260929 20:56:04.932313 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.931709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49354)
0929 20:56:04.931975 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.932105 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 20:56:04.933545 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.933124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49364)
0929 20:56:04.933324 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.933405 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260929 20:56:04.933789 13855 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40373: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40373: connect: Connection refused (error 111)
I20260929 20:56:04.939157 13936 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40373
I20260929 20:56:04.939179 14002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40373 every 8 connection(s)
I20260929 20:56:04.940510 13936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-2/data/info.pb
I20260929 20:56:04.944254 14003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:04.946395 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 13936
I20260929 20:56:04.946571 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715169435170-388-0/minicluster-data/master-2/wal/instance
I20260929 20:56:04.947171 388 external_mini_cluster.cc:1428] Setting key 137a52da56669de977af7fdf18bf3e3a
I20260929 20:56:04.951808 14003 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:56:04 dist-test-slave-m9g1 krb5kdc[13775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 20:56:04 dist-test-slave-m9g1 krb5kdc[13775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:56:04.968721 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.954554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53506 (local address 127.0.97.62:39953)
0929 20:56:04.954751 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:04.954757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:04.955443 (+ 686us) server_negotiation.cc:407] Connection header received
0929 20:56:04.955577 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:04.955581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:04.955640 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:04.955718 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:04.956704 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.957446 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.958862 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.959503 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.959634 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:04.962699 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:04.962705 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:04.962709 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:04.962747 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:04.965678 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:04.966487 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:04.966492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:04.966496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:04.966662 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:04.966997 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:04.967002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:04.967005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:04.967324 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:04.967472 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:04.968133 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 20:56:04.968379 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:56:04.969568 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.949035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.0.97.60:40373)
0929 20:56:04.949855 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:04.949870 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:04.949913 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:56:04.951500 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:04.951523 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:04.951927 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:04.952251 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:04.953901 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.955639 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.956985 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.957622 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.957809 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:04.961923 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:04.961932 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:04.961943 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:04.961982 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:04.965201 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:04.965901 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:04.965911 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:04.965914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:04.966068 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:04.966519 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:04.966524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:04.966527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:04.967223 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:04.967514 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:04.967940 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 20:56:04.968388 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":372,"threads_started":1}
I20260929 20:56:04.969611 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.954557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39953 (local address 127.0.0.1:53506)
0929 20:56:04.955284 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.955347 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:04.955481 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:04.955790 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:04.955796 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:04.956306 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:04.956548 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:04.956557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.957586 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.957591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:04.958724 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:04.958735 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.959686 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.959690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:04.959814 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:04.960382 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:04.960402 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:04.962514 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:04.965812 (+ 3298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:04.965824 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:04.965838 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:04.966379 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:04.966761 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:04.966765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:04.966767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:04.966885 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:04.967438 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:04.967446 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:04.967686 (+ 240us) client_negotiation.cc:769] Sending connection context
0929 20:56:04.967975 (+ 289us) client_negotiation.cc:240] Negotiation successful
0929 20:56:04.968552 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":153,"thread_start_us":363,"threads_started":1}
I20260929 20:56:04.971074 14003 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:56:04 dist-test-slave-m9g1 krb5kdc[13775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 29 20:56:04 dist-test-slave-m9g1 krb5kdc[13775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:56:04.985786 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.971798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41147 (local address 127.0.0.1:60094)
0929 20:56:04.971959 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.971977 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:04.972076 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:04.972694 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:04.972700 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:04.973147 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:04.973291 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:04.973299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.974609 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.974615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:04.975706 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:04.975718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.976616 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.976622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:04.976810 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:04.977448 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:04.977468 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:04.979546 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:04.982720 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:04.982727 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:04.982731 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:04.983280 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:04.983769 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:04.983774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:04.983778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:04.983956 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:04.984801 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:04.984806 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:04.984990 (+ 184us) client_negotiation.cc:769] Sending connection context
0929 20:56:04.985145 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 20:56:04.985441 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 20:56:04.985881 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.972067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60094 (local address 127.0.97.61:41147)
0929 20:56:04.972263 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:04.972267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:04.972283 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:56:04.972333 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:04.972336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:04.972413 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:04.972511 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:04.973469 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.974469 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.975846 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.976484 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.976598 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:04.979746 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:04.979752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:04.979755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:04.979791 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:04.982532 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:04.983438 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:04.983445 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:04.983449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:04.983636 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:04.984285 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:04.984291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:04.984294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:04.984663 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:04.984816 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:04.985273 (+ 457us) server_negotiation.cc:299] Negotiation successful
0929 20:56:04.985506 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:56:04.986937 14003 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:56:04.989326 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.972477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.0.97.62:39953)
0929 20:56:04.972636 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:04.972641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:04.972657 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:56:04.972754 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:04.972758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:04.972824 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:04.972919 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:04.974171 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.975145 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.976576 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.977160 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.977260 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:04.982254 (+ 4994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:04.982263 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:04.982268 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:04.982316 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:04.985781 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:04.987004 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:04.987011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:04.987014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:04.987189 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:04.987601 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:04.987607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:04.987610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:04.987979 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:04.988266 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:04.988783 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 20:56:04.989010 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 20:56:04 dist-test-slave-m9g1 krb5kdc[13775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 20:56:04 dist-test-slave-m9g1 krb5kdc[13775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:56:04.999946 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.988092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49388 (local address 127.0.97.60:40373)
0929 20:56:04.988237 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:04.988242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:04.988258 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:56:04.988302 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:04.988307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:04.988373 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:04.988464 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:04.989360 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.990360 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.991595 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.992023 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.992126 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:04.995005 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:04.995012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:04.995016 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:04.995052 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:04.997510 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:04.998157 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:04.998162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:04.998166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:04.998330 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:04.998710 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:04.998715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:04.998717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:04.998978 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:04.999101 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:04.999483 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 20:56:04.999670 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":49}
I20260929 20:56:05.000217 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.987759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49388)
0929 20:56:04.987972 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:04.987991 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:04.988088 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:04.988549 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:04.988554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:04.989041 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:04.989220 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:04.989232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.990470 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.990475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:04.991463 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:04.991475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.992148 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.992153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:04.992354 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:04.992893 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:04.992909 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:04.994814 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:04.997643 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:04.997648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:04.997651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:04.998046 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:04.998500 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:04.998503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:04.998506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:04.998615 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:04.999082 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:04.999086 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:04.999227 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 20:56:04.999391 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 20:56:04.999835 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":111}
I20260929 20:56:05.004995 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:04.990967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60110 (local address 127.0.97.61:41147)
0929 20:56:04.991146 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:04.991152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:04.991170 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:56:04.991276 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:04.991282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:04.991348 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:04.991434 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:04.992585 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.993276 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.994504 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:04.994927 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:04.995050 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:04.999268 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:04.999275 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:04.999279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:04.999318 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:05.002495 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:05.003153 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:05.003159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:05.003163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:05.003294 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:05.003662 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:05.003668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:05.003671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:05.003949 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:05.004191 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:05.004459 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 20:56:05.004685 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:56:05.005182 14003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f: Bootstrap starting.
I20260929 20:56:05.008997 14003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:05.009661 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:56:05.010121 14003 log.cc:824] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:05.013090 14003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f: No bootstrap required, opened a new log
I20260929 20:56:05.022116 14003 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } } peers { permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } } peers { permanent_uuid: "d8d15958932b423c956bf7c19c256e0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } }
I20260929 20:56:05.022686 14003 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:05.022804 14003 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d15958932b423c956bf7c19c256e0f, State: Initialized, Role: FOLLOWER
I20260929 20:56:05.023646 14003 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } } peers { permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } } peers { permanent_uuid: "d8d15958932b423c956bf7c19c256e0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } }
I20260929 20:56:05.027943 14016 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } } peers { permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } } peers { permanent_uuid: "d8d15958932b423c956bf7c19c256e0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } } }
I20260929 20:56:05.028424 14016 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:56:05.030387 14003 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:56:05.037106 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:05.018368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.0.97.62:39953)
0929 20:56:05.018556 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:05.018562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:05.018706 (+ 144us) server_negotiation.cc:407] Connection header received
0929 20:56:05.019015 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:05.019021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:05.019107 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:05.019178 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:05.021643 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:05.022802 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:05.025042 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:05.025666 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:05.025766 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:05.029616 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:05.029622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:05.029627 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:05.029669 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:05.033081 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:05.034051 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:05.034056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:05.034060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:05.034255 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:05.034754 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:05.034759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:05.034763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:05.035167 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:05.035390 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:05.036505 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0929 20:56:05.036723 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:56:05.037545 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:05.018029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.0.97.60:40373)
0929 20:56:05.018229 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:05.018235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:05.019239 (+ 1004us) server_negotiation.cc:407] Connection header received
0929 20:56:05.019470 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:05.019475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:05.019545 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:05.019615 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:05.021811 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:05.022852 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:05.025655 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:05.026462 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:05.026558 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:05.030243 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:05.030249 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:05.030253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:05.030290 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:05.034047 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:05.035003 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:05.035009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:05.035013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:05.035198 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:05.035628 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:05.035634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:05.035637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:05.036042 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:05.036291 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:05.036902 (+ 611us) server_negotiation.cc:299] Negotiation successful
0929 20:56:05.037146 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 20:56:05.040843 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:05.019387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60122 (local address 127.0.97.61:41147)
0929 20:56:05.019555 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:05.019560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:05.023856 (+ 4296us) server_negotiation.cc:407] Connection header received
0929 20:56:05.024095 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:05.024101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:05.024179 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:05.024243 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:05.026481 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:05.027456 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:05.029588 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:05.030231 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:05.030327 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:05.034076 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:05.034082 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:05.034087 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:05.034126 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:05.037544 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:05.038467 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:05.038472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:05.038475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:05.038635 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:05.039042 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:05.039047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:05.039051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:05.039443 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:05.039671 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:05.040232 (+ 561us) server_negotiation.cc:299] Negotiation successful
0929 20:56:05.040501 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 20:56:05.048552 14027 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:56:05.048671 14027 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:56:05.045105 13936 master_runner.cc:186] Error getting master registration for 127.0.97.62:39953: OK, instance_id { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" instance_seqno: 1790715363973509 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:56:05.056133 13936 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:56:05.110997 13855 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } attempt: 4
Sep 29 20:56:05 dist-test-slave-m9g1 krb5kdc[13775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 20:56:05.124923 14016 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:56:05.125294 14016 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } } peers { permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } } peers { permanent_uuid: "d8d15958932b423c956bf7c19c256e0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } }
I20260929 20:56:05.125684 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:05.111550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49404)
0929 20:56:05.111741 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:05.111762 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:05.111869 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:05.112277 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:05.112281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:05.112682 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:05.112873 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:05.112888 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:05.114327 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:05.114331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:05.115262 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:05.115275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:05.116194 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:05.116198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:05.116365 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:05.116997 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:05.117015 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:05.119388 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:05.123099 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:05.123105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:05.123109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:05.123644 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:05.124004 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:05.124009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:05.124013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:05.124192 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:05.124655 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:05.124660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:05.124829 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 20:56:05.124982 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 20:56:05.125311 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 20:56:05.125883 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:05.111630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49404 (local address 127.0.97.60:40373)
0929 20:56:05.111838 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:05.111844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:05.111865 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:56:05.111995 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:05.112001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:05.112103 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:05.112211 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:05.113025 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:05.114168 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:05.115398 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:05.116071 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:05.116195 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:05.119586 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:05.119593 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:05.119597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:05.119636 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:05.122978 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:05.123745 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:05.123748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:05.123751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:05.123892 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:05.124288 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:05.124291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:05.124293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:05.124552 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:05.124675 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:05.125179 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 20:56:05.125407 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:56:05.127009 14016 leader_election.cc:288] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9216a20083ef4edaa0190753b702dfe3 (127.0.97.62:39953), ea685a1b25f14d918281710e627a686a (127.0.97.61:41147)
I20260929 20:56:05.131567 13855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3: Bootstrap starting.
I20260929 20:56:05.135501 13855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:05.136804 13855 log.cc:824] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:05.140198 13855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3: No bootstrap required, opened a new log
I20260929 20:56:05.150821 13855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } } peers { permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } } peers { permanent_uuid: "d8d15958932b423c956bf7c19c256e0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } }
I20260929 20:56:05.151206 13855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:05.151328 13855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9216a20083ef4edaa0190753b702dfe3, State: Initialized, Role: FOLLOWER
I20260929 20:56:05.151978 13855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } } peers { permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } } peers { permanent_uuid: "d8d15958932b423c956bf7c19c256e0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } }
I20260929 20:56:05.153721 14030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } } peers { permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } } peers { permanent_uuid: "d8d15958932b423c956bf7c19c256e0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } } }
I20260929 20:56:05.154441 14030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:56:05.154934 13855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:56:05.155006 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8d15958932b423c956bf7c19c256e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9216a20083ef4edaa0190753b702dfe3" is_pre_election: true
I20260929 20:56:05.155584 13830 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d15958932b423c956bf7c19c256e0f in term 0.
I20260929 20:56:05.156225 13952 leader_election.cc:302] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9216a20083ef4edaa0190753b702dfe3, d8d15958932b423c956bf7c19c256e0f; no voters:
I20260929 20:56:05.156695 14016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:56:05.156867 14016 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:56:05.156976 14016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:05.160686 14016 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } } peers { permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } } peers { permanent_uuid: "d8d15958932b423c956bf7c19c256e0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } }
I20260929 20:56:05.162286 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8d15958932b423c956bf7c19c256e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9216a20083ef4edaa0190753b702dfe3"
I20260929 20:56:05.162511 13830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:05.162569 14016 leader_election.cc:288] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [CANDIDATE]: Term 1 election: Requested vote from peers 9216a20083ef4edaa0190753b702dfe3 (127.0.97.62:39953), ea685a1b25f14d918281710e627a686a (127.0.97.61:41147)
I20260929 20:56:05.165746 13830 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d15958932b423c956bf7c19c256e0f in term 1.
I20260929 20:56:05.166265 13952 leader_election.cc:302] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9216a20083ef4edaa0190753b702dfe3, d8d15958932b423c956bf7c19c256e0f; no voters:
I20260929 20:56:05.166734 14016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [term 1 FOLLOWER]: Leader election won for term 1
W20260929 20:56:05.174067 13957 tablet.cc:2369] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:56:05.179139 14016 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [term 1 LEADER]: Becoming Leader. State: Replica: d8d15958932b423c956bf7c19c256e0f, State: Running, Role: LEADER
I20260929 20:56:05.179730 14016 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } } peers { permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } } peers { permanent_uuid: "d8d15958932b423c956bf7c19c256e0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } }
I20260929 20:56:05.186247 13929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } attempt: 5
I20260929 20:56:05.189260 14036 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8d15958932b423c956bf7c19c256e0f. Latest consensus state: current_term: 1 leader_uuid: "d8d15958932b423c956bf7c19c256e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } } peers { permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } } peers { permanent_uuid: "d8d15958932b423c956bf7c19c256e0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } } }
I20260929 20:56:05.189702 14036 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [sys.catalog]: This master's current role is: LEADER
I20260929 20:56:05.180474 13788 master_runner.cc:186] Error getting master registration for 127.0.97.61:41147: OK, instance_id { permanent_uuid: "ea685a1b25f14d918281710e627a686a" instance_seqno: 1790715364447368 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:56:05.195003 13788 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 29 20:56:05 dist-test-slave-m9g1 krb5kdc[13775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260929 20:56:05.199750 14042 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:56:05.199915 14042 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:56:05.206209 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:05.186857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40373 (local address 127.0.0.1:49420)
0929 20:56:05.187033 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:05.187052 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:05.187163 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:05.187665 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:05.187670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:05.188373 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:05.188578 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:05.188591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:05.190282 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:05.190288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:05.191675 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:05.191690 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:05.192893 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:05.192899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:05.193101 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:05.193780 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:05.193826 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:05.197103 (+ 3277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:05.202580 (+ 5477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:05.202585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:05.202589 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:05.203106 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:05.203878 (+ 772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:05.203883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:05.203887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:05.204048 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:05.205190 (+ 1142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:05.205195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:05.205359 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 20:56:05.205502 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 20:56:05.205790 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 20:56:05.206763 14045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:56:05.207454 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:05.187157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49420 (local address 127.0.97.60:40373)
0929 20:56:05.187347 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:05.187353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:05.187373 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:56:05.187430 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:05.187435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:05.187510 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:05.187624 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:05.188757 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:05.189949 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:05.191866 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:05.192537 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:05.192647 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:05.197310 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:05.197317 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:05.197321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:05.197358 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:05.202355 (+ 4997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:05.203389 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:05.203396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:05.203399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:05.203677 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:05.204243 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:05.204248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:05.204283 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:05.204982 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:05.206662 (+ 1680us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:05.206808 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 20:56:05.207055 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:56:05.209033 14045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:56:05.213251 13929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a: Bootstrap starting.
W20260929 20:56:05.213577 13809 tablet.cc:2369] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:56:05.223137 13929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:05.225597 13929 log.cc:824] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:05.239013 13929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a: No bootstrap required, opened a new log
I20260929 20:56:05.242887 13830 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3 [term 1 FOLLOWER]: Refusing update from remote peer d8d15958932b423c956bf7c19c256e0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:56:05.243983 14036 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:56:05.256672 13929 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } } peers { permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } } peers { permanent_uuid: "d8d15958932b423c956bf7c19c256e0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } }
I20260929 20:56:05.257320 13929 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:05.257512 13929 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea685a1b25f14d918281710e627a686a, State: Initialized, Role: FOLLOWER
I20260929 20:56:05.258486 13929 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } } peers { permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } } peers { permanent_uuid: "d8d15958932b423c956bf7c19c256e0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } }
I20260929 20:56:05.268213 14030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8d15958932b423c956bf7c19c256e0f. Latest consensus state: current_term: 1 leader_uuid: "d8d15958932b423c956bf7c19c256e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } } peers { permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } } peers { permanent_uuid: "d8d15958932b423c956bf7c19c256e0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } } }
I20260929 20:56:05.268510 14030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:56:05.271401 14049 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } } peers { permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } } peers { permanent_uuid: "d8d15958932b423c956bf7c19c256e0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } } }
I20260929 20:56:05.273934 14046 mvcc.cc:205] Tried to move back new op lower bound from 7334770136017260544 to 7334770135781122048. Current Snapshot: MvccSnapshot[applied={T|T < 7334770136017260544}]
I20260929 20:56:05.273689 13904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8d15958932b423c956bf7c19c256e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea685a1b25f14d918281710e627a686a" is_pre_election: true
I20260929 20:56:05.274114 13929 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:56:05.274086 13902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:05.274281 13903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8d15958932b423c956bf7c19c256e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea685a1b25f14d918281710e627a686a"
I20260929 20:56:05.276432 14049 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:56:05.277787 14048 mvcc.cc:205] Tried to move back new op lower bound from 7334770136017260544 to 7334770135781122048. Current Snapshot: MvccSnapshot[applied={T|T < 7334770136017260544}]
I20260929 20:56:05.279173 13902 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a [term 1 FOLLOWER]: Refusing update from remote peer d8d15958932b423c956bf7c19c256e0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:56:05.280126 14016 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:56:05.295681 14016 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8d15958932b423c956bf7c19c256e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } } peers { permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } } peers { permanent_uuid: "d8d15958932b423c956bf7c19c256e0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } } }
I20260929 20:56:05.296052 14016 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [sys.catalog]: This master's current role is: LEADER
I20260929 20:56:05.299479 14045 catalog_manager.cc:1347] Generated new cluster ID: c9c26e1f9f3a47d58f73ea0dc9a11d54
I20260929 20:56:05.299597 14045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:56:05.302479 14030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8d15958932b423c956bf7c19c256e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } } peers { permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } } peers { permanent_uuid: "d8d15958932b423c956bf7c19c256e0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } } }
I20260929 20:56:05.302734 14030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:56:05.304374 14054 mvcc.cc:205] Tried to move back new op lower bound from 7334770136017260544 to 7334770135781122048. Current Snapshot: MvccSnapshot[applied={T|T < 7334770136017260544}]
I20260929 20:56:05.306419 14049 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8d15958932b423c956bf7c19c256e0f. Latest consensus state: current_term: 1 leader_uuid: "d8d15958932b423c956bf7c19c256e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } } peers { permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } } peers { permanent_uuid: "d8d15958932b423c956bf7c19c256e0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } } }
I20260929 20:56:05.306741 14049 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:56:05.306388 14056 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8d15958932b423c956bf7c19c256e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } } peers { permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } } peers { permanent_uuid: "d8d15958932b423c956bf7c19c256e0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } } }
I20260929 20:56:05.306972 14056 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:56:05.348284 14036 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8d15958932b423c956bf7c19c256e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9216a20083ef4edaa0190753b702dfe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39953 } } peers { permanent_uuid: "ea685a1b25f14d918281710e627a686a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41147 } } peers { permanent_uuid: "d8d15958932b423c956bf7c19c256e0f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40373 } } }
I20260929 20:56:05.348593 14036 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f [sys.catalog]: This master's current role is: LEADER
I20260929 20:56:05.352480 14045 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:56:05.354475 14045 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:56:05.371030 14045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8d15958932b423c956bf7c19c256e0f: Generated new TSK 0
I20260929 20:56:05.372027 14045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:56:05.381474 14068 catalog_manager.cc:1259] Loaded cluster ID: c9c26e1f9f3a47d58f73ea0dc9a11d54
I20260929 20:56:05.381568 14068 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a: loading cluster ID for follower catalog manager: success
I20260929 20:56:05.385454 14068 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a: acquiring CA information for follower catalog manager: success
I20260929 20:56:05.387892 14068 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ea685a1b25f14d918281710e627a686a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:56:06.201910 14042 catalog_manager.cc:1259] Loaded cluster ID: c9c26e1f9f3a47d58f73ea0dc9a11d54
I20260929 20:56:06.202008 14042 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3: loading cluster ID for follower catalog manager: success
I20260929 20:56:06.204468 14042 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3: acquiring CA information for follower catalog manager: success
I20260929 20:56:06.206274 14042 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9216a20083ef4edaa0190753b702dfe3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:56:06.515949 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 13788
I20260929 20:56:06.517244 13850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:06.629657 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 13788
I20260929 20:56:06.647401 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 13862
I20260929 20:56:06.648419 13924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:06.754740 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 13862
I20260929 20:56:06.770750 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 13936
I20260929 20:56:06.771759 13998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:06.878198 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 13936
2026-09-29T20:56:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4564 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:56:06 dist-test-slave-m9g1 krb5kdc[14080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:56:06 dist-test-slave-m9g1 krb5kdc[14080](info): set up 2 sockets
Sep 29 20:56:06 dist-test-slave-m9g1 krb5kdc[14080](info): commencing operation
krb5kdc: starting...
W20260929 20:56:07.878374 388 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:56:08 dist-test-slave-m9g1 krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:56:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:56:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:56:08.099927 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35853
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:56:08.414146 14093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:08.414456 14093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:08.414585 14093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:08.421891 14093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:56:08.422010 14093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:08.422065 14093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:08.422154 14093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:56:08.422235 14093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:56:08.432220 14093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:08.433544 14093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:08.434904 14093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:08.446098 14098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:08.446120 14099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:08.446553 14101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:08.447037 14093 server_base.cc:1034] running on GCE node
I20260929 20:56:08.447989 14093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:08.449563 14093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:08.450758 14093 hybrid_clock.cc:648] HybridClock initialized: now 1790715368450703 us; error 40 us; skew 500 ppm
I20260929 20:56:08.451113 14093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:08 dist-test-slave-m9g1 krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:08.454025 14093 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:56:08.455497 14093 webserver.cc:460] Webserver started at http://127.0.97.62:40351/ using document root <none> and password file <none>
I20260929 20:56:08.456161 14093 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:08.456280 14093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:08.456561 14093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:56:08.459484 14093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "08c59d3089ed41e3bb67810bfd855578"
format_stamp: "Formatted at 2026-09-29 20:56:08 on dist-test-slave-m9g1"
server_key: "5d7432192bcb1a394856aa2354fab145"
server_key_iv: "f6436771149f2e3c4cc7ad5cfd3a76bb"
server_key_version: "encryptionkey@0"
I20260929 20:56:08.460189 14093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "08c59d3089ed41e3bb67810bfd855578"
format_stamp: "Formatted at 2026-09-29 20:56:08 on dist-test-slave-m9g1"
server_key: "5d7432192bcb1a394856aa2354fab145"
server_key_iv: "f6436771149f2e3c4cc7ad5cfd3a76bb"
server_key_version: "encryptionkey@0"
I20260929 20:56:08.465162 14093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:56:08.468477 14108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:08.469813 14093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:56:08.469965 14093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "08c59d3089ed41e3bb67810bfd855578"
format_stamp: "Formatted at 2026-09-29 20:56:08 on dist-test-slave-m9g1"
server_key: "5d7432192bcb1a394856aa2354fab145"
server_key_iv: "f6436771149f2e3c4cc7ad5cfd3a76bb"
server_key_version: "encryptionkey@0"
I20260929 20:56:08.470132 14093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:08.491777 14093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:08.496004 14093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:08.496281 14093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:08.513178 14093 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35853
I20260929 20:56:08.513198 14159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35853 every 8 connection(s)
I20260929 20:56:08.514564 14093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:56:08.518298 14160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:08.520749 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 14093
I20260929 20:56:08.520943 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:56:08.521544 388 external_mini_cluster.cc:1428] Setting key 775e183301e13013627c80097ed09b6f
I20260929 20:56:08.529826 14160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578: Bootstrap starting.
I20260929 20:56:08.533468 14160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:08.534746 14160 log.cc:824] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:56:08 dist-test-slave-m9g1 krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:56:08.537856 14160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578: No bootstrap required, opened a new log
I20260929 20:56:08.543929 14163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:08.523309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43116 (local address 127.0.97.62:35853)
0929 20:56:08.524040 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:08.524051 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:08.524094 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:56:08.525697 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:08.525718 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:08.526113 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:08.526494 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:08.528221 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:08.529485 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:08.531251 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:08.531937 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:08.532063 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:08.536939 (+ 4876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:08.536946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:08.536960 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:08.536994 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:08.540006 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:08.540838 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:08.540844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:08.540849 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:08.541012 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:08.541379 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:08.541382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:08.541384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:08.541890 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:08.542102 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:08.542545 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 20:56:08.542903 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":317,"threads_started":1}
I20260929 20:56:08.546864 14160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c59d3089ed41e3bb67810bfd855578" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35853 } }
I20260929 20:56:08.547338 14160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:08.547446 14160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c59d3089ed41e3bb67810bfd855578, State: Initialized, Role: FOLLOWER
I20260929 20:56:08.547961 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:56:08.548033 14160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c59d3089ed41e3bb67810bfd855578" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35853 } }
I20260929 20:56:08.548290 14160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:56:08.548419 14160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:56:08.548575 14160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:08.550710 14160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c59d3089ed41e3bb67810bfd855578" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35853 } }
I20260929 20:56:08.551231 14160 leader_election.cc:302] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08c59d3089ed41e3bb67810bfd855578; no voters:
I20260929 20:56:08.551749 14160 leader_election.cc:288] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:56:08.551887 14165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:56:08.552495 14165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578 [term 1 LEADER]: Becoming Leader. State: Replica: 08c59d3089ed41e3bb67810bfd855578, State: Running, Role: LEADER
I20260929 20:56:08.552976 14165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c59d3089ed41e3bb67810bfd855578" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35853 } }
I20260929 20:56:08.553591 14160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:56:08.555675 14166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08c59d3089ed41e3bb67810bfd855578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c59d3089ed41e3bb67810bfd855578" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35853 } } }
I20260929 20:56:08.555794 14167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08c59d3089ed41e3bb67810bfd855578. Latest consensus state: current_term: 1 leader_uuid: "08c59d3089ed41e3bb67810bfd855578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c59d3089ed41e3bb67810bfd855578" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35853 } } }
I20260929 20:56:08.556052 14167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578 [sys.catalog]: This master's current role is: LEADER
I20260929 20:56:08.556052 14166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578 [sys.catalog]: This master's current role is: LEADER
I20260929 20:56:08.557646 14173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:56:08.562646 14173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:56:08.570869 14173 catalog_manager.cc:1347] Generated new cluster ID: 0c901565409d457a95f5f5dbfd249459
I20260929 20:56:08.570998 14173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:56:08.619997 14173 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:56:08.621315 14173 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:56:08.632251 14173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08c59d3089ed41e3bb67810bfd855578: Generated new TSK 0
I20260929 20:56:08.633111 14173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:56:08.659113 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 14093
I20260929 20:56:08.660213 14155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:08.754748 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 14093
2026-09-29T20:56:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1872 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:56:08 dist-test-slave-m9g1 krb5kdc[14188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:56:08 dist-test-slave-m9g1 krb5kdc[14188](info): set up 2 sockets
Sep 29 20:56:08 dist-test-slave-m9g1 krb5kdc[14188](info): commencing operation
krb5kdc: starting...
W20260929 20:56:09.766366 388 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:56:09 dist-test-slave-m9g1 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:56:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:56:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:56:09.985848 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42065
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:56:10.311395 14201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:10.311693 14201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:10.311813 14201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:10.319187 14201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:56:10.319304 14201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:10.319386 14201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:10.319468 14201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:56:10.319527 14201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:56:10.329607 14201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:10.330981 14201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:10.332340 14201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:10.343997 14206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:10.344022 14209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:10.344061 14207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:10.345194 14201 server_base.cc:1034] running on GCE node
I20260929 20:56:10.346021 14201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:10.347472 14201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:10.348671 14201 hybrid_clock.cc:648] HybridClock initialized: now 1790715370348625 us; error 35 us; skew 500 ppm
I20260929 20:56:10.349001 14201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:10 dist-test-slave-m9g1 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:10.354780 14201 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:56:10.356335 14201 webserver.cc:460] Webserver started at http://127.0.97.62:33889/ using document root <none> and password file <none>
I20260929 20:56:10.357048 14201 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:10.357146 14201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:10.357410 14201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:56:10.360451 14201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "d7ce0a71a1d343d6b3656da6ac4baf59"
format_stamp: "Formatted at 2026-09-29 20:56:10 on dist-test-slave-m9g1"
server_key: "0a4f7f99467529bdfe7ce10370610136"
server_key_iv: "7f3dace5656679fd14a5f3bae5d7f1d1"
server_key_version: "encryptionkey@0"
I20260929 20:56:10.361191 14201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "d7ce0a71a1d343d6b3656da6ac4baf59"
format_stamp: "Formatted at 2026-09-29 20:56:10 on dist-test-slave-m9g1"
server_key: "0a4f7f99467529bdfe7ce10370610136"
server_key_iv: "7f3dace5656679fd14a5f3bae5d7f1d1"
server_key_version: "encryptionkey@0"
I20260929 20:56:10.366398 14201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 20:56:10.369833 14216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:10.371238 14201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:56:10.371418 14201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "d7ce0a71a1d343d6b3656da6ac4baf59"
format_stamp: "Formatted at 2026-09-29 20:56:10 on dist-test-slave-m9g1"
server_key: "0a4f7f99467529bdfe7ce10370610136"
server_key_iv: "7f3dace5656679fd14a5f3bae5d7f1d1"
server_key_version: "encryptionkey@0"
I20260929 20:56:10.371598 14201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:10.395738 14201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:10.399978 14201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:10.400238 14201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:10.417811 14201 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42065
I20260929 20:56:10.417833 14267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42065 every 8 connection(s)
I20260929 20:56:10.419178 14201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:56:10.422955 14268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:10.427383 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 14201
I20260929 20:56:10.427600 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:56:10.428444 388 external_mini_cluster.cc:1428] Setting key 206555b36c5f0397d456cb295a4b2b1c
I20260929 20:56:10.434655 14268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59: Bootstrap starting.
I20260929 20:56:10.438248 14268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:10.439386 14268 log.cc:824] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:10.442018 14268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59: No bootstrap required, opened a new log
Sep 29 20:56:10 dist-test-slave-m9g1 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:56:10.452286 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:10.430692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53196 (local address 127.0.97.62:42065)
0929 20:56:10.431474 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:10.431486 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:10.431530 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:56:10.433384 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:10.433411 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:10.433829 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:10.434159 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:10.435818 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:10.437163 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:10.438873 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:10.439586 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:10.439706 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:10.443606 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:10.443616 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:10.443638 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:10.443691 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:10.447710 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:10.448446 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:10.448456 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:10.448465 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:10.448677 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:10.449050 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:10.449056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:10.449060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:10.449783 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:10.450095 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:10.450501 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 20:56:10.450968 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":363,"threads_started":1}
I20260929 20:56:10.452065 14268 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42065 } }
I20260929 20:56:10.452494 14268 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:10.452639 14268 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ce0a71a1d343d6b3656da6ac4baf59, State: Initialized, Role: FOLLOWER
I20260929 20:56:10.453341 14268 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42065 } }
I20260929 20:56:10.453625 14268 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:56:10.453764 14268 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:56:10.453980 14268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:10.456821 14268 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42065 } }
I20260929 20:56:10.457283 14268 leader_election.cc:302] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7ce0a71a1d343d6b3656da6ac4baf59; no voters:
I20260929 20:56:10.457836 14268 leader_election.cc:288] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:56:10.457969 14273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:56:10.458596 14273 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 1 LEADER]: Becoming Leader. State: Replica: d7ce0a71a1d343d6b3656da6ac4baf59, State: Running, Role: LEADER
I20260929 20:56:10.459117 14273 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42065 } }
I20260929 20:56:10.459743 14268 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:56:10.462615 14275 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7ce0a71a1d343d6b3656da6ac4baf59. Latest consensus state: current_term: 1 leader_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42065 } } }
I20260929 20:56:10.462630 14274 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42065 } } }
I20260929 20:56:10.463002 14275 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [sys.catalog]: This master's current role is: LEADER
I20260929 20:56:10.463002 14274 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [sys.catalog]: This master's current role is: LEADER
I20260929 20:56:10.463822 14282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:56:10.469304 14282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:56:10.480643 14282 catalog_manager.cc:1347] Generated new cluster ID: 435fd7e57fd946648322b67d3af39e63
I20260929 20:56:10.480767 14282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:56:10.491041 14282 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:56:10.492362 14282 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:56:10.500597 14282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59: Generated new TSK 0
I20260929 20:56:10.501405 14282 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:56:10.559477 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42065
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:56:10.905099 14294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:10.905388 14294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:10.905505 14294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:10.912775 14294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:10.912921 14294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:10.913076 14294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:56:10.922962 14294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:10.924268 14294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:10.925582 14294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:10.938167 14300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:10.938097 14299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:10.938133 14302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:10.939142 14294 server_base.cc:1034] running on GCE node
I20260929 20:56:10.939771 14294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:10.940846 14294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:10.942119 14294 hybrid_clock.cc:648] HybridClock initialized: now 1790715370942034 us; error 73 us; skew 500 ppm
I20260929 20:56:10.942499 14294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:10 dist-test-slave-m9g1 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:10.946130 14294 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:56:10.947966 14294 webserver.cc:460] Webserver started at http://127.0.97.1:46451/ using document root <none> and password file <none>
I20260929 20:56:10.948738 14294 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:10.948859 14294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:10.949165 14294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:56:10.952416 14294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "89f226196eee47df894280b06f688e19"
format_stamp: "Formatted at 2026-09-29 20:56:10 on dist-test-slave-m9g1"
server_key: "0d423f27789b6d0c53b28cff79169ee8"
server_key_iv: "f7c9812520a5f6be89dd4e5ae30c2c26"
server_key_version: "encryptionkey@0"
I20260929 20:56:10.953321 14294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "89f226196eee47df894280b06f688e19"
format_stamp: "Formatted at 2026-09-29 20:56:10 on dist-test-slave-m9g1"
server_key: "0d423f27789b6d0c53b28cff79169ee8"
server_key_iv: "f7c9812520a5f6be89dd4e5ae30c2c26"
server_key_version: "encryptionkey@0"
I20260929 20:56:10.958981 14294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:56:10.962654 14309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:10.964062 14294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:56:10.964210 14294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "89f226196eee47df894280b06f688e19"
format_stamp: "Formatted at 2026-09-29 20:56:10 on dist-test-slave-m9g1"
server_key: "0d423f27789b6d0c53b28cff79169ee8"
server_key_iv: "f7c9812520a5f6be89dd4e5ae30c2c26"
server_key_version: "encryptionkey@0"
I20260929 20:56:10.964437 14294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:10.991259 14294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:10.995700 14294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:10.996034 14294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:10.996995 14294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:56:10.998325 14294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:56:10.998401 14294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:10.998518 14294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:56:10.998569 14294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:11.029028 14294 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44095
I20260929 20:56:11.029080 14421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44095 every 8 connection(s)
I20260929 20:56:11.030386 14294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:56:11.034337 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 14294
I20260929 20:56:11.034544 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:56:11.035353 388 external_mini_cluster.cc:1428] Setting key 2768150d52b147267998a6d5533cb4c2
Sep 29 20:56:11 dist-test-slave-m9g1 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:56:11.051039 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:11.033046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39199 (local address 127.0.97.62:42065)
0929 20:56:11.033511 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:11.033517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:11.034705 (+ 1188us) server_negotiation.cc:407] Connection header received
0929 20:56:11.036794 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:11.036800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:11.036886 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:11.036981 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:11.038667 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:11.039408 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:11.041056 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:11.041481 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:11.041597 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:11.045169 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:11.045174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:11.045178 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:11.045213 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:11.047773 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:11.048579 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:11.048585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:11.048587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:11.048715 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:11.049161 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:11.049165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:11.049167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:11.049448 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:11.049579 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:11.050420 (+ 841us) server_negotiation.cc:299] Negotiation successful
0929 20:56:11.050665 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":217,"threads_started":1}
I20260929 20:56:11.052073 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:11.033661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:39199)
0929 20:56:11.034523 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:11.034581 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:11.036502 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:11.037253 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:11.037263 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:11.037814 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:11.038481 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:11.038501 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:11.039576 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:11.039580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:11.040887 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:11.040904 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:11.041715 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:11.041723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:11.041968 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:11.042679 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:11.042702 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:11.044919 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:11.047943 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:11.047954 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:11.047973 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:11.048435 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:11.048824 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:11.048838 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:11.048842 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:11.049041 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:11.049574 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:11.049582 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:11.049954 (+ 372us) client_negotiation.cc:769] Sending connection context
0929 20:56:11.050288 (+ 334us) client_negotiation.cc:240] Negotiation successful
0929 20:56:11.050874 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":342,"threads_started":1}
I20260929 20:56:11.053849 14422 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42065
I20260929 20:56:11.054283 14422 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:56:11.055210 14422 heartbeater.cc:511] Master 127.0.97.62:42065 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 20:56:11.057785 14233 ts_manager.cc:194] Registered new tserver with Master: 89f226196eee47df894280b06f688e19 (127.0.97.1:44095)
I20260929 20:56:11.059378 14233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:56:11.085863 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42065
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:56:11.399425 14428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:11.399732 14428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:11.399870 14428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:11.407183 14428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:11.407305 14428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:11.407467 14428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:56:11.417235 14428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:11.418543 14428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:11.419823 14428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:11.433840 14434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:11.433892 14433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:11.434302 14436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:11.435153 14428 server_base.cc:1034] running on GCE node
I20260929 20:56:11.435711 14428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:11.436646 14428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:11.437839 14428 hybrid_clock.cc:648] HybridClock initialized: now 1790715371437788 us; error 45 us; skew 500 ppm
I20260929 20:56:11.438140 14428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:11 dist-test-slave-m9g1 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:11.441033 14428 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:56:11.442548 14428 webserver.cc:460] Webserver started at http://127.0.97.2:38513/ using document root <none> and password file <none>
I20260929 20:56:11.443198 14428 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:11.443308 14428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:11.443581 14428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:56:11.446439 14428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "07463aaa39ac4d4db7ff63a6657e7ee1"
format_stamp: "Formatted at 2026-09-29 20:56:11 on dist-test-slave-m9g1"
server_key: "855385f3890720b51872262db871a6a4"
server_key_iv: "4c5f85ece3f3cc271b83700c60805dfb"
server_key_version: "encryptionkey@0"
I20260929 20:56:11.447175 14428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "07463aaa39ac4d4db7ff63a6657e7ee1"
format_stamp: "Formatted at 2026-09-29 20:56:11 on dist-test-slave-m9g1"
server_key: "855385f3890720b51872262db871a6a4"
server_key_iv: "4c5f85ece3f3cc271b83700c60805dfb"
server_key_version: "encryptionkey@0"
I20260929 20:56:11.452148 14428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:56:11.455590 14443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:11.457414 14428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 20:56:11.457576 14428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "07463aaa39ac4d4db7ff63a6657e7ee1"
format_stamp: "Formatted at 2026-09-29 20:56:11 on dist-test-slave-m9g1"
server_key: "855385f3890720b51872262db871a6a4"
server_key_iv: "4c5f85ece3f3cc271b83700c60805dfb"
server_key_version: "encryptionkey@0"
I20260929 20:56:11.457782 14428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:11.470629 14428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:11.474771 14428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:11.475033 14428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:11.476004 14428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:56:11.477294 14428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:56:11.477370 14428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:11.477478 14428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:56:11.477535 14428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:11.506829 14428 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43715
I20260929 20:56:11.506865 14555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43715 every 8 connection(s)
I20260929 20:56:11.508085 14428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:56:11.516952 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 14428
I20260929 20:56:11.517144 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:56:11.517692 388 external_mini_cluster.cc:1428] Setting key af79afd9a32d0a9f32580c07925b8c8e
Sep 29 20:56:11 dist-test-slave-m9g1 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:56:11.529016 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:11.510626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53597 (local address 127.0.97.62:42065)
0929 20:56:11.510858 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:11.510863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:11.512084 (+ 1221us) server_negotiation.cc:407] Connection header received
0929 20:56:11.513866 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:11.513871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:11.513946 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:11.514049 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:11.515715 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:11.516412 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:11.518644 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:11.519122 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:11.519217 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:11.523161 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:11.523166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:11.523169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:11.523207 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:11.525805 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:11.526644 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:11.526649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:11.526651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:11.526791 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:11.527228 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:11.527232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:11.527235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:11.527519 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:11.527651 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:11.528453 (+ 802us) server_negotiation.cc:299] Negotiation successful
0929 20:56:11.528690 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 20:56:11.530025 14558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:11.511148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:53597)
0929 20:56:11.511890 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:11.511947 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:11.513584 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:11.514366 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:11.514375 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:11.514939 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:11.515544 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:11.515562 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:11.516592 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:11.516598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:11.518499 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:11.518515 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:11.519345 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:11.519352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:11.519621 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:11.520323 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:11.520357 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:11.522886 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:11.525964 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:11.525974 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:11.525993 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:11.526506 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:11.526906 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:11.526911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:11.526915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:11.527111 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:11.527643 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:11.527652 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:11.528027 (+ 375us) client_negotiation.cc:769] Sending connection context
0929 20:56:11.528297 (+ 270us) client_negotiation.cc:240] Negotiation successful
0929 20:56:11.528860 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":291,"threads_started":1}
I20260929 20:56:11.531545 14556 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42065
I20260929 20:56:11.531837 14556 heartbeater.cc:464] Registering TS with master...
I20260929 20:56:11.532578 14556 heartbeater.cc:511] Master 127.0.97.62:42065 requested a full tablet report, sending...
I20260929 20:56:11.534027 14233 ts_manager.cc:194] Registered new tserver with Master: 07463aaa39ac4d4db7ff63a6657e7ee1 (127.0.97.2:43715)
I20260929 20:56:11.534839 14233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53597
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:56:11.566861 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42065
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:56:11.882352 14561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:11.882637 14561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:11.882730 14561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:11.890014 14561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:11.890134 14561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:11.890288 14561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:56:11.900202 14561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:11.901522 14561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:11.903105 14561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:11.916157 14567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:11.916183 14569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:11.916203 14566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:11.917140 14561 server_base.cc:1034] running on GCE node
I20260929 20:56:11.917845 14561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:11.918916 14561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:11.920151 14561 hybrid_clock.cc:648] HybridClock initialized: now 1790715371920100 us; error 45 us; skew 500 ppm
I20260929 20:56:11.920472 14561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:11 dist-test-slave-m9g1 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:11.923693 14561 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:56:11.925278 14561 webserver.cc:460] Webserver started at http://127.0.97.3:39597/ using document root <none> and password file <none>
I20260929 20:56:11.925988 14561 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:11.926110 14561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:11.926440 14561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:56:11.929545 14561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "d6e98474a4b0491684729581e8230622"
format_stamp: "Formatted at 2026-09-29 20:56:11 on dist-test-slave-m9g1"
server_key: "c19dc47e9f83adcd42bac4b9d781f463"
server_key_iv: "9a1659d989496bc30b03a3606d2d13d0"
server_key_version: "encryptionkey@0"
I20260929 20:56:11.930392 14561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d6e98474a4b0491684729581e8230622"
format_stamp: "Formatted at 2026-09-29 20:56:11 on dist-test-slave-m9g1"
server_key: "c19dc47e9f83adcd42bac4b9d781f463"
server_key_iv: "9a1659d989496bc30b03a3606d2d13d0"
server_key_version: "encryptionkey@0"
I20260929 20:56:11.935812 14561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260929 20:56:11.939452 14576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:11.940900 14561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:56:11.941076 14561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "d6e98474a4b0491684729581e8230622"
format_stamp: "Formatted at 2026-09-29 20:56:11 on dist-test-slave-m9g1"
server_key: "c19dc47e9f83adcd42bac4b9d781f463"
server_key_iv: "9a1659d989496bc30b03a3606d2d13d0"
server_key_version: "encryptionkey@0"
I20260929 20:56:11.941269 14561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:11.971529 14561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:11.975891 14561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:11.976169 14561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:11.977121 14561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:56:11.978514 14561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:56:11.978590 14561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:11.978701 14561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:56:11.978763 14561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:12.007905 14561 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38667
I20260929 20:56:12.007953 14688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38667 every 8 connection(s)
I20260929 20:56:12.009209 14561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:56:12.018922 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 14561
I20260929 20:56:12.019111 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:56:12.019652 388 external_mini_cluster.cc:1428] Setting key ebb7ee54b5a987e76890ee93fdabde49
Sep 29 20:56:12 dist-test-slave-m9g1 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:56:12.030253 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:12.011874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39145 (local address 127.0.97.62:42065)
0929 20:56:12.012064 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:12.012068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:12.013499 (+ 1431us) server_negotiation.cc:407] Connection header received
0929 20:56:12.015386 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:12.015393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:12.015478 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:12.015577 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:12.017164 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.017938 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.020073 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.020582 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.020704 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:12.024237 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:12.024243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:12.024246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:12.024282 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:12.026918 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:12.027674 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:12.027679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:12.027682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:12.027835 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:12.028310 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:12.028315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:12.028317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:12.028723 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:12.028875 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:12.029574 (+ 699us) server_negotiation.cc:299] Negotiation successful
0929 20:56:12.029826 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:56:12.030987 14691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:12.012506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:39145)
0929 20:56:12.013311 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:12.013350 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:12.015083 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:12.015826 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:12.015835 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:12.016393 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:12.016977 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:12.016995 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.018101 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.018105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:12.019920 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:12.019936 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.020807 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.020815 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:12.021053 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:12.021707 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:12.021733 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:12.024032 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:12.027073 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:12.027084 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:12.027100 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:12.027532 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:12.027952 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:12.027957 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:12.027960 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:12.028186 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:12.028840 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:12.028846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:12.029174 (+ 328us) client_negotiation.cc:769] Sending connection context
0929 20:56:12.029400 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 20:56:12.029810 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":323,"threads_started":1}
I20260929 20:56:12.032509 14689 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42065
I20260929 20:56:12.032826 14689 heartbeater.cc:464] Registering TS with master...
I20260929 20:56:12.033722 14689 heartbeater.cc:511] Master 127.0.97.62:42065 requested a full tablet report, sending...
I20260929 20:56:12.035362 14232 ts_manager.cc:194] Registered new tserver with Master: d6e98474a4b0491684729581e8230622 (127.0.97.3:38667)
I20260929 20:56:12.036226 14232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39145
I20260929 20:56:12.048002 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:56:12.062729 14422 heartbeater.cc:503] Master 127.0.97.62:42065 was elected leader, sending a full tablet report...
I20260929 20:56:12.065840 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:12.052453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53206 (local address 127.0.97.62:42065)
0929 20:56:12.052628 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:12.052634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:12.053029 (+ 395us) server_negotiation.cc:407] Connection header received
0929 20:56:12.053209 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:12.053214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:12.053280 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:12.053361 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:12.054813 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.055538 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.056871 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.057296 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.057405 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:12.059811 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:12.059817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:12.059820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:12.059857 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:12.062818 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:12.063578 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:12.063583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:12.063585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:12.063711 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:12.064128 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:12.064132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:12.064134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:12.064654 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:12.064892 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:12.065248 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 20:56:12.065519 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 20:56:12.070099 14232 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:56:12.077077 14232 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:56:12.103725 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:12.089436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60332 (local address 127.0.97.1:44095)
0929 20:56:12.091463 (+ 2027us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:12.091473 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:12.091517 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:56:12.091621 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:12.091631 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:12.091916 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:12.094013 (+ 2097us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:12.094293 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.097733 (+ 3440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.101317 (+ 3584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.102538 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.102653 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:12.103032 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:12.103233 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 20:56:12.103392 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1857,"thread_start_us":1700,"threads_started":1}
I20260929 20:56:12.103760 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:12.089081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44095 (local address 127.0.0.1:60332)
0929 20:56:12.090640 (+ 1559us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:12.090694 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:12.090870 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:12.093251 (+ 2381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:12.093260 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:12.093296 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:12.093752 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:12.093770 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.098593 (+ 4823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.098599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:12.101116 (+ 2517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:12.101142 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.102736 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.102742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:12.102967 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:12.103054 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 20:56:12.103185 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 20:56:12.103377 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1242,"thread_start_us":8880,"threads_started":1}
I20260929 20:56:12.106470 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:12.089285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38667 (local address 127.0.0.1:33766)
0929 20:56:12.098305 (+ 9020us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:12.098327 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:12.098442 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:12.099363 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:12.099368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:12.099400 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:12.099863 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:12.099880 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.101655 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.101660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:12.103658 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:12.103670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.105702 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.105709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:12.105867 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:12.105900 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:56:12.105981 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 20:56:12.106120 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8843,"thread_start_us":878,"threads_started":1}
I20260929 20:56:12.106937 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:12.090025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33766 (local address 127.0.97.3:38667)
0929 20:56:12.098699 (+ 8674us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:12.098709 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:12.098745 (+ 36us) server_negotiation.cc:407] Connection header received
0929 20:56:12.098850 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:12.098858 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:12.099143 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:12.099365 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:12.100083 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.101515 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.103847 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.105212 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.105343 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:12.105560 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:12.106440 (+ 880us) server_negotiation.cc:299] Negotiation successful
0929 20:56:12.106610 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8483,"thread_start_us":8323,"threads_started":1}
I20260929 20:56:12.108111 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:12.091563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43715 (local address 127.0.0.1:48528)
0929 20:56:12.100316 (+ 8753us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:12.100353 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:12.100457 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:12.101343 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:12.101347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:12.101371 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:12.101739 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:12.101756 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.103600 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.103605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:12.105544 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:12.105557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.107340 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.107346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:12.107506 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:12.107529 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:56:12.107608 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 20:56:12.107731 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8632,"mutex_wait_us":1,"thread_start_us":8504,"threads_started":1}
I20260929 20:56:12.108127 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:12.092117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48528 (local address 127.0.97.2:43715)
0929 20:56:12.099084 (+ 6967us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:12.099093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:12.100583 (+ 1490us) server_negotiation.cc:407] Connection header received
0929 20:56:12.100685 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:12.100693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:12.100924 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:12.101111 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:12.101920 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.103425 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.105735 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.107154 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.107275 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:12.107455 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:12.107667 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 20:56:12.107828 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6797,"thread_start_us":6668,"threads_started":1}
I20260929 20:56:12.108695 14357 tablet_service.cc:1467] Processing CreateTablet for tablet 72e92ccbd42d4e2c90cf9e3f60be0c4e (DEFAULT_TABLE table=test-table [id=4710c91f6b49474c97bdc081b148af24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:56:12.110304 14357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e92ccbd42d4e2c90cf9e3f60be0c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:12.110814 14624 tablet_service.cc:1467] Processing CreateTablet for tablet 72e92ccbd42d4e2c90cf9e3f60be0c4e (DEFAULT_TABLE table=test-table [id=4710c91f6b49474c97bdc081b148af24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:56:12.111032 14491 tablet_service.cc:1467] Processing CreateTablet for tablet 72e92ccbd42d4e2c90cf9e3f60be0c4e (DEFAULT_TABLE table=test-table [id=4710c91f6b49474c97bdc081b148af24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:56:12.112309 14624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e92ccbd42d4e2c90cf9e3f60be0c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:12.112310 14491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e92ccbd42d4e2c90cf9e3f60be0c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:12.123239 14707 tablet_bootstrap.cc:492] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1: Bootstrap starting.
I20260929 20:56:12.126492 14709 tablet_bootstrap.cc:492] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622: Bootstrap starting.
I20260929 20:56:12.126921 14707 tablet_bootstrap.cc:654] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:12.128648 14707 log.cc:824] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:12.129967 14709 tablet_bootstrap.cc:654] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:12.130746 14708 tablet_bootstrap.cc:492] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19: Bootstrap starting.
I20260929 20:56:12.131295 14709 log.cc:824] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:12.131978 14707 tablet_bootstrap.cc:492] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1: No bootstrap required, opened a new log
I20260929 20:56:12.132233 14707 ts_tablet_manager.cc:1397] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260929 20:56:12.133658 14708 tablet_bootstrap.cc:654] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:12.134405 14709 tablet_bootstrap.cc:492] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622: No bootstrap required, opened a new log
I20260929 20:56:12.134668 14709 ts_tablet_manager.cc:1397] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 20:56:12.134764 14708 log.cc:824] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:12.138149 14708 tablet_bootstrap.cc:492] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19: No bootstrap required, opened a new log
I20260929 20:56:12.138466 14708 ts_tablet_manager.cc:1397] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260929 20:56:12.142262 14707 raft_consensus.cc:348] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } }
I20260929 20:56:12.142762 14707 raft_consensus.cc:374] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:12.142899 14707 raft_consensus.cc:729] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07463aaa39ac4d4db7ff63a6657e7ee1, State: Initialized, Role: FOLLOWER
I20260929 20:56:12.143553 14707 consensus_queue.cc:260] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } }
I20260929 20:56:12.144699 14556 heartbeater.cc:503] Master 127.0.97.62:42065 was elected leader, sending a full tablet report...
I20260929 20:56:12.144833 14709 raft_consensus.cc:348] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } }
I20260929 20:56:12.145309 14709 raft_consensus.cc:374] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:12.145412 14707 ts_tablet_manager.cc:1428] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260929 20:56:12.145407 14709 raft_consensus.cc:729] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e98474a4b0491684729581e8230622, State: Initialized, Role: FOLLOWER
I20260929 20:56:12.146005 14709 consensus_queue.cc:260] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } }
I20260929 20:56:12.147034 14689 heartbeater.cc:503] Master 127.0.97.62:42065 was elected leader, sending a full tablet report...
I20260929 20:56:12.146775 14708 raft_consensus.cc:348] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } }
I20260929 20:56:12.147171 14708 raft_consensus.cc:374] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:12.147272 14708 raft_consensus.cc:729] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f226196eee47df894280b06f688e19, State: Initialized, Role: FOLLOWER
I20260929 20:56:12.147922 14708 consensus_queue.cc:260] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } }
I20260929 20:56:12.148198 14709 ts_tablet_manager.cc:1428] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260929 20:56:12.149693 14708 ts_tablet_manager.cc:1428] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19: Time spent starting tablet: real 0.011s user 0.007s sys 0.002s
I20260929 20:56:12.189356 14714 raft_consensus.cc:482] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:56:12.189656 14714 raft_consensus.cc:504] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } }
I20260929 20:56:12.191314 14714 leader_election.cc:288] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f226196eee47df894280b06f688e19 (127.0.97.1:44095), 07463aaa39ac4d4db7ff63a6657e7ee1 (127.0.97.2:43715)
I20260929 20:56:12.197502 14691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:12.191657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44095 (local address 127.0.97.3:47139)
0929 20:56:12.192102 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:12.192118 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:12.192244 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:12.192546 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:12.192550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:12.192577 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:12.192777 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:12.192785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.193949 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.193952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:12.195721 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:12.195736 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.196984 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.196988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:12.197121 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:12.197145 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:56:12.197212 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 20:56:12.197300 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":41}
I20260929 20:56:12.197628 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:12.191755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47139 (local address 127.0.97.1:44095)
0929 20:56:12.191957 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:12.191963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:12.192199 (+ 236us) server_negotiation.cc:407] Connection header received
0929 20:56:12.192383 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:12.192388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:12.192444 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:12.192523 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:12.192905 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.193843 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.195873 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.196836 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.196961 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:12.197060 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:12.197250 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 20:56:12.197381 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:56:12.198590 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:12.192185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43715 (local address 127.0.97.3:47439)
0929 20:56:12.192842 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:12.192856 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:12.192955 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:12.193252 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:12.193254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:12.193271 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:12.193527 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:12.193534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.194648 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.194654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:12.196323 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:12.196334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.197942 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.197948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:12.198109 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:12.198138 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 20:56:12.198252 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 20:56:12.198336 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":3,"thread_start_us":340,"threads_started":1}
I20260929 20:56:12.198683 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:12.191785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47439 (local address 127.0.97.2:43715)
0929 20:56:12.191976 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:12.191981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:12.192923 (+ 942us) server_negotiation.cc:407] Connection header received
0929 20:56:12.193079 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:12.193084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:12.193156 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:12.193242 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:12.193670 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.194498 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.196479 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.197602 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.197742 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:12.197827 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:12.198286 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 20:56:12.198408 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:56:12.198701 14377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e92ccbd42d4e2c90cf9e3f60be0c4e" candidate_uuid: "d6e98474a4b0491684729581e8230622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f226196eee47df894280b06f688e19" is_pre_election: true
I20260929 20:56:12.199352 14377 raft_consensus.cc:2454] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e98474a4b0491684729581e8230622 in term 0.
I20260929 20:56:12.199645 14511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e92ccbd42d4e2c90cf9e3f60be0c4e" candidate_uuid: "d6e98474a4b0491684729581e8230622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" is_pre_election: true
I20260929 20:56:12.200062 14580 leader_election.cc:302] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f226196eee47df894280b06f688e19, d6e98474a4b0491684729581e8230622; no voters:
I20260929 20:56:12.200140 14511 raft_consensus.cc:2454] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e98474a4b0491684729581e8230622 in term 0.
I20260929 20:56:12.200451 14714 raft_consensus.cc:2781] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:56:12.200605 14714 raft_consensus.cc:482] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:56:12.200716 14714 raft_consensus.cc:3037] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:12.202921 14714 raft_consensus.cc:504] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } }
I20260929 20:56:12.203475 14714 leader_election.cc:288] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [CANDIDATE]: Term 1 election: Requested vote from peers 89f226196eee47df894280b06f688e19 (127.0.97.1:44095), 07463aaa39ac4d4db7ff63a6657e7ee1 (127.0.97.2:43715)
I20260929 20:56:12.203907 14377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e92ccbd42d4e2c90cf9e3f60be0c4e" candidate_uuid: "d6e98474a4b0491684729581e8230622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f226196eee47df894280b06f688e19"
I20260929 20:56:12.203923 14511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e92ccbd42d4e2c90cf9e3f60be0c4e" candidate_uuid: "d6e98474a4b0491684729581e8230622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1"
I20260929 20:56:12.204080 14511 raft_consensus.cc:3037] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:12.204102 14377 raft_consensus.cc:3037] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:12.206147 14511 raft_consensus.cc:2454] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e98474a4b0491684729581e8230622 in term 1.
I20260929 20:56:12.206158 14377 raft_consensus.cc:2454] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e98474a4b0491684729581e8230622 in term 1.
I20260929 20:56:12.206542 14577 leader_election.cc:302] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07463aaa39ac4d4db7ff63a6657e7ee1, d6e98474a4b0491684729581e8230622; no voters:
I20260929 20:56:12.206820 14714 raft_consensus.cc:2781] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:56:12.207227 14714 raft_consensus.cc:686] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [term 1 LEADER]: Becoming Leader. State: Replica: d6e98474a4b0491684729581e8230622, State: Running, Role: LEADER
I20260929 20:56:12.207655 14714 consensus_queue.cc:237] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } }
I20260929 20:56:12.212538 14232 catalog_manager.cc:5495] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e98474a4b0491684729581e8230622 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d6e98474a4b0491684729581e8230622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } health_report { overall_health: UNKNOWN } } }
I20260929 20:56:12.254487 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:12.249293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33768 (local address 127.0.97.3:38667)
0929 20:56:12.249453 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:12.249458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:12.249476 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:56:12.249650 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:12.249654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:12.249730 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:12.249910 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:56:12.250282 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.250992 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.252508 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.252908 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.253001 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:12.253472 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:56:12.253988 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:56:12.254099 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:12.254216 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 20:56:12.254289 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 20:56:12.260134 14557 tablet.cc:2369] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:56:12.260867 14690 tablet.cc:2369] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:56:12.262719 14377 raft_consensus.cc:1261] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19 [term 1 FOLLOWER]: Refusing update from remote peer d6e98474a4b0491684729581e8230622: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:56:12.262742 14511 raft_consensus.cc:1261] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1 [term 1 FOLLOWER]: Refusing update from remote peer d6e98474a4b0491684729581e8230622: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:56:12.263707 14717 consensus_queue.cc:1035] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:56:12.264057 14714 consensus_queue.cc:1035] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:56:12.292591 14231 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:56:12.292870 14231 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:56:12.296478 14231 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59: Sending DeleteTablet for 3 replicas of tablet 72e92ccbd42d4e2c90cf9e3f60be0c4e
I20260929 20:56:12.297616 14624 tablet_service.cc:1514] Processing DeleteTablet for tablet 72e92ccbd42d4e2c90cf9e3f60be0c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:56:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33766
I20260929 20:56:12.297684 14357 tablet_service.cc:1514] Processing DeleteTablet for tablet 72e92ccbd42d4e2c90cf9e3f60be0c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:56:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60332
I20260929 20:56:12.297832 14491 tablet_service.cc:1514] Processing DeleteTablet for tablet 72e92ccbd42d4e2c90cf9e3f60be0c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:56:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48528
I20260929 20:56:12.298789 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 14294
I20260929 20:56:12.310701 14732 tablet_replica.cc:330] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1: stopping tablet replica
I20260929 20:56:12.311261 14732 raft_consensus.cc:2229] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:56:12.310127 14731 tablet_replica.cc:330] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19: stopping tablet replica
I20260929 20:56:12.311571 14730 tablet_replica.cc:330] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622: stopping tablet replica
I20260929 20:56:12.312042 14731 raft_consensus.cc:2229] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:56:12.312075 14730 raft_consensus.cc:2229] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:56:12.312742 14732 raft_consensus.cc:2258] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:56:12.312757 14730 raft_consensus.cc:2258] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:56:12.313673 14731 raft_consensus.cc:2258] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:56:12.315399 14730 ts_tablet_manager.cc:1905] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:56:12.317315 14731 ts_tablet_manager.cc:1905] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:56:12.318679 14732 ts_tablet_manager.cc:1905] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 20:56:12 dist-test-slave-m9g1 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:56:12.323133 14730 ts_tablet_manager.cc:1918] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:56:12.323370 14730 log.cc:1196] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/72e92ccbd42d4e2c90cf9e3f60be0c4e
I20260929 20:56:12.323951 14730 ts_tablet_manager.cc:1939] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P d6e98474a4b0491684729581e8230622: Deleting consensus metadata
I20260929 20:56:12.325924 14219 catalog_manager.cc:4841] TS d6e98474a4b0491684729581e8230622 (127.0.97.3:38667): tablet 72e92ccbd42d4e2c90cf9e3f60be0c4e (table test-table [id=4710c91f6b49474c97bdc081b148af24]) successfully deleted
I20260929 20:56:12.326606 14731 ts_tablet_manager.cc:1918] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:56:12.326787 14731 log.cc:1196] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/72e92ccbd42d4e2c90cf9e3f60be0c4e
I20260929 20:56:12.327386 14731 ts_tablet_manager.cc:1939] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 89f226196eee47df894280b06f688e19: Deleting consensus metadata
I20260929 20:56:12.329483 14220 catalog_manager.cc:4841] TS 89f226196eee47df894280b06f688e19 (127.0.97.1:44095): tablet 72e92ccbd42d4e2c90cf9e3f60be0c4e (table test-table [id=4710c91f6b49474c97bdc081b148af24]) successfully deleted
I20260929 20:56:12.330464 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:12.299984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60336 (local address 127.0.97.1:44095)
0929 20:56:12.300161 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:12.300168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:12.310525 (+ 10357us) server_negotiation.cc:407] Connection header received
0929 20:56:12.310733 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:12.310739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:12.310837 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:12.310947 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:12.313009 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.314143 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.316245 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.316851 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.316979 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:12.322538 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:12.322553 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:12.322571 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:12.322620 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:12.326917 (+ 4297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:12.327904 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:12.327914 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:12.327917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:12.328085 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:12.328602 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:12.328607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:12.328610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:12.329082 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:12.329355 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:12.329998 (+ 643us) server_negotiation.cc:299] Negotiation successful
0929 20:56:12.330172 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:56:12.331718 14417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:12.333492 14732 ts_tablet_manager.cc:1918] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:56:12.333832 14732 log.cc:1196] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/72e92ccbd42d4e2c90cf9e3f60be0c4e
I20260929 20:56:12.334697 14732 ts_tablet_manager.cc:1939] T 72e92ccbd42d4e2c90cf9e3f60be0c4e P 07463aaa39ac4d4db7ff63a6657e7ee1: Deleting consensus metadata
I20260929 20:56:12.336652 14217 catalog_manager.cc:4841] TS 07463aaa39ac4d4db7ff63a6657e7ee1 (127.0.97.2:43715): tablet 72e92ccbd42d4e2c90cf9e3f60be0c4e (table test-table [id=4710c91f6b49474c97bdc081b148af24]) successfully deleted
I20260929 20:56:12.431064 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 14294
I20260929 20:56:12.452198 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 14428
Sep 29 20:56:12 dist-test-slave-m9g1 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:56:12.469370 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:12.453006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48542 (local address 127.0.97.2:43715)
0929 20:56:12.453209 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:12.453215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:12.453237 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:56:12.453326 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:12.453332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:12.453409 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:12.453508 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:12.454911 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.456024 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.457347 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.457972 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.458105 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:12.463403 (+ 5298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:12.463419 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:12.463435 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:12.463481 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:12.466315 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:12.467240 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:12.467248 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:12.467251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:12.467397 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:12.467800 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:12.467804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:12.467806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:12.468214 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:12.468488 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:12.468898 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 20:56:12.469095 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 20:56:12.470059 14551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:12.558656 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 14428
I20260929 20:56:12.574280 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 14561
Sep 29 20:56:12 dist-test-slave-m9g1 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:56:12.593176 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:12.575651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33772 (local address 127.0.97.3:38667)
0929 20:56:12.575858 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:12.575864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:12.575883 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:56:12.575962 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:12.575967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:12.576072 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:12.576198 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:12.577769 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.579210 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.580768 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:12.581464 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:12.581642 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:12.586044 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:12.586065 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:12.586088 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:12.586142 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:12.590155 (+ 4013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:12.590911 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:12.590926 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:12.590929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:12.591145 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:12.591497 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:12.591502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:12.591505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:12.591989 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:12.592318 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:12.592514 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 20:56:12.592752 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:56:12.593966 14684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:12.717439 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 14561
I20260929 20:56:12.736440 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 14201
I20260929 20:56:12.737535 14263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:12.848624 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 14201
I20260929 20:56:12.866024 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42065
--webserver_interface=127.0.97.62
--webserver_port=33889
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:56:13.184907 14738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:13.185209 14738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:13.185341 14738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:13.192603 14738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:56:13.192729 14738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:13.192801 14738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:13.192881 14738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:56:13.192940 14738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:56:13.202694 14738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:13.203981 14738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:13.205302 14738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:13.226792 14744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:13.226890 14747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:13.226850 14745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:13.227753 14738 server_base.cc:1034] running on GCE node
I20260929 20:56:13.228641 14738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:13.230262 14738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:13.231484 14738 hybrid_clock.cc:648] HybridClock initialized: now 1790715373231422 us; error 49 us; skew 500 ppm
I20260929 20:56:13.231861 14738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:13 dist-test-slave-m9g1 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:13.235195 14738 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:56:13.236850 14738 webserver.cc:460] Webserver started at http://127.0.97.62:33889/ using document root <none> and password file <none>
I20260929 20:56:13.237586 14738 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:13.237720 14738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:13.243678 14738 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 20:56:13.246716 14754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:13.248073 14738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 20:56:13.248245 14738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "d7ce0a71a1d343d6b3656da6ac4baf59"
format_stamp: "Formatted at 2026-09-29 20:56:10 on dist-test-slave-m9g1"
server_key: "0a4f7f99467529bdfe7ce10370610136"
server_key_iv: "7f3dace5656679fd14a5f3bae5d7f1d1"
server_key_version: "encryptionkey@0"
I20260929 20:56:13.249312 14738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:13.265436 14738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:13.269678 14738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:13.269909 14738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:13.286638 14738 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42065
I20260929 20:56:13.286662 14805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42065 every 8 connection(s)
I20260929 20:56:13.288133 14738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:56:13.295118 14806 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:56:13.297375 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 14738
I20260929 20:56:13.298337 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44095
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42065
--webserver_port=46451
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:56:13.299057 14806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59: Bootstrap starting.
I20260929 20:56:13.319964 14806 log.cc:824] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:13.332340 14806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:56:13.332867 14806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59: Bootstrap complete.
I20260929 20:56:13.341925 14806 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42065 } }
I20260929 20:56:13.342875 14806 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ce0a71a1d343d6b3656da6ac4baf59, State: Initialized, Role: FOLLOWER
I20260929 20:56:13.343565 14806 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42065 } }
I20260929 20:56:13.343797 14806 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:56:13.343905 14806 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:56:13.344066 14806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:56:13.347921 14806 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42065 } }
I20260929 20:56:13.348459 14806 leader_election.cc:302] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7ce0a71a1d343d6b3656da6ac4baf59; no voters:
I20260929 20:56:13.349063 14806 leader_election.cc:288] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:56:13.349195 14810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:56:13.350349 14810 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 2 LEADER]: Becoming Leader. State: Replica: d7ce0a71a1d343d6b3656da6ac4baf59, State: Running, Role: LEADER
I20260929 20:56:13.351056 14810 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42065 } }
I20260929 20:56:13.351536 14806 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:56:13.353729 14812 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7ce0a71a1d343d6b3656da6ac4baf59. Latest consensus state: current_term: 2 leader_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42065 } } }
I20260929 20:56:13.353781 14811 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42065 } } }
I20260929 20:56:13.354135 14812 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [sys.catalog]: This master's current role is: LEADER
I20260929 20:56:13.354138 14811 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [sys.catalog]: This master's current role is: LEADER
I20260929 20:56:13.355381 14815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:56:13.369365 14815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:56:13.373188 14815 catalog_manager.cc:1259] Loaded cluster ID: 435fd7e57fd946648322b67d3af39e63
I20260929 20:56:13.373385 14815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:56:13.377113 14815 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 20:56:13.400122 14815 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 20:56:13.423578 14815 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:56:13.425782 14815 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59: Loaded TSK: 0
I20260929 20:56:13.427865 14815 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 20:56:13.648764 14808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:13.649042 14808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:13.649166 14808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:13.656728 14808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:13.656839 14808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:13.656981 14808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:56:13.666985 14808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=46451
--tserver_master_addrs=127.0.97.62:42065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:13.668360 14808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:13.669762 14808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:13.684618 14834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:13.684618 14833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:13.685356 14836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:13.685869 14808 server_base.cc:1034] running on GCE node
I20260929 20:56:13.686543 14808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:13.687583 14808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:13.688794 14808 hybrid_clock.cc:648] HybridClock initialized: now 1790715373688743 us; error 44 us; skew 500 ppm
I20260929 20:56:13.689105 14808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:13 dist-test-slave-m9g1 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:13.692374 14808 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:56:13.693907 14808 webserver.cc:460] Webserver started at http://127.0.97.1:46451/ using document root <none> and password file <none>
I20260929 20:56:13.694630 14808 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:13.694764 14808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:13.699976 14808 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 20:56:13.702607 14843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:13.703900 14808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:56:13.704043 14808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "89f226196eee47df894280b06f688e19"
format_stamp: "Formatted at 2026-09-29 20:56:10 on dist-test-slave-m9g1"
server_key: "0d423f27789b6d0c53b28cff79169ee8"
server_key_iv: "f7c9812520a5f6be89dd4e5ae30c2c26"
server_key_version: "encryptionkey@0"
I20260929 20:56:13.704967 14808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:13.719342 14808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:13.723562 14808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:13.723831 14808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:13.724848 14808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:56:13.726279 14808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:56:13.726372 14808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:13.726475 14808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:56:13.726549 14808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:13.760318 14808 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44095
I20260929 20:56:13.760385 14955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44095 every 8 connection(s)
I20260929 20:56:13.761844 14808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:56:13.762490 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 14808
I20260929 20:56:13.763218 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:43715
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42065
--webserver_port=38513
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:56:13 dist-test-slave-m9g1 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:56:13.788617 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:13.765077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:46861)
0929 20:56:13.765923 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:13.765970 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:13.767752 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:13.769497 (+ 1745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:13.769508 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:13.770311 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:13.771018 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:13.771065 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:13.773333 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:13.773339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:13.774655 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:13.774668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:13.776057 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:13.776063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:13.776275 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:13.776968 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:13.776993 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:13.779492 (+ 2499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:13.784132 (+ 4640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:13.784140 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:13.784154 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:13.784636 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:13.785186 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:13.785189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:13.785192 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:13.785373 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:13.786398 (+ 1025us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:13.786406 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:13.786820 (+ 414us) client_negotiation.cc:769] Sending connection context
0929 20:56:13.787125 (+ 305us) client_negotiation.cc:240] Negotiation successful
0929 20:56:13.787549 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":332,"threads_started":1}
I20260929 20:56:13.788965 14958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:13.764872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46861 (local address 127.0.97.62:42065)
0929 20:56:13.765479 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:13.765496 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:13.766124 (+ 628us) server_negotiation.cc:407] Connection header received
0929 20:56:13.768378 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:13.768396 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:13.768880 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:13.769316 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:13.771314 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:13.773155 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:13.774794 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:13.775809 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:13.775980 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:13.779892 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:13.779904 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:13.779929 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:13.779983 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:13.783924 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:13.784807 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:13.784817 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:13.784826 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:13.785069 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:13.785488 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:13.785493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:13.785496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:13.786251 (+ 755us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:13.786444 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:13.787389 (+ 945us) server_negotiation.cc:299] Negotiation successful
0929 20:56:13.787826 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":268,"threads_started":1}
I20260929 20:56:13.791903 14956 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42065
I20260929 20:56:13.792191 14956 heartbeater.cc:464] Registering TS with master...
I20260929 20:56:13.792905 14956 heartbeater.cc:511] Master 127.0.97.62:42065 requested a full tablet report, sending...
I20260929 20:56:13.795257 14771 ts_manager.cc:194] Registered new tserver with Master: 89f226196eee47df894280b06f688e19 (127.0.97.1:44095)
I20260929 20:56:13.797276 14771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46861
W20260929 20:56:14.083033 14960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:14.083323 14960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:14.083416 14960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:14.090571 14960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:14.090692 14960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:14.090814 14960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:56:14.100256 14960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:43715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=38513
--tserver_master_addrs=127.0.97.62:42065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:14.101496 14960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:14.102773 14960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:14.115438 14965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:14.115456 14968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:14.115517 14966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:14.116780 14960 server_base.cc:1034] running on GCE node
I20260929 20:56:14.117398 14960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:14.118376 14960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:14.119617 14960 hybrid_clock.cc:648] HybridClock initialized: now 1790715374119533 us; error 51 us; skew 500 ppm
I20260929 20:56:14.119942 14960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:14 dist-test-slave-m9g1 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:14.123000 14960 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:56:14.124367 14960 webserver.cc:460] Webserver started at http://127.0.97.2:38513/ using document root <none> and password file <none>
I20260929 20:56:14.125030 14960 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:14.125154 14960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:14.130405 14960 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:56:14.132939 14975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:14.134260 14960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:56:14.134411 14960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "07463aaa39ac4d4db7ff63a6657e7ee1"
format_stamp: "Formatted at 2026-09-29 20:56:11 on dist-test-slave-m9g1"
server_key: "855385f3890720b51872262db871a6a4"
server_key_iv: "4c5f85ece3f3cc271b83700c60805dfb"
server_key_version: "encryptionkey@0"
I20260929 20:56:14.135363 14960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:14.147003 14960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:14.150807 14960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:14.151057 14960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:14.152004 14960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:56:14.153700 14960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:56:14.153797 14960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:14.153899 14960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:56:14.153971 14960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:14.182515 14960 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43715
I20260929 20:56:14.182576 15087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43715 every 8 connection(s)
I20260929 20:56:14.183879 14960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:56:14.186528 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 14960
I20260929 20:56:14.187074 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38667
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42065
--webserver_port=39597
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:56:14 dist-test-slave-m9g1 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:56:14.202747 14958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:14.186262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56843 (local address 127.0.97.62:42065)
0929 20:56:14.186459 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:14.186465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:14.187721 (+ 1256us) server_negotiation.cc:407] Connection header received
0929 20:56:14.189500 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:14.189506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:14.189590 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:14.189695 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:14.191136 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.192071 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.193548 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.193998 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.194106 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:14.197253 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:14.197258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:14.197261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:14.197293 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:14.199794 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:14.200492 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:14.200496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:14.200498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:14.200620 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:14.201042 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:14.201046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:14.201048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:14.201325 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:14.201458 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:14.202125 (+ 667us) server_negotiation.cc:299] Negotiation successful
0929 20:56:14.202385 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:56:14.203666 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:14.186784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:56843)
0929 20:56:14.187554 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:14.187598 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:14.189238 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:14.189856 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:14.189864 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:14.190423 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:14.190973 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:14.190991 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.192203 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.192206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:14.193407 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:14.193418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.194239 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.194247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:14.194451 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:14.195049 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:14.195070 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:14.197047 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:14.199936 (+ 2889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:14.199945 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:14.199962 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:14.200369 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:14.200727 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:14.200732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:14.200736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:14.200929 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:14.201450 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:14.201458 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:14.201764 (+ 306us) client_negotiation.cc:769] Sending connection context
0929 20:56:14.201994 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 20:56:14.202530 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":301,"threads_started":1}
I20260929 20:56:14.205088 15088 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42065
I20260929 20:56:14.205365 15088 heartbeater.cc:464] Registering TS with master...
I20260929 20:56:14.206027 15088 heartbeater.cc:511] Master 127.0.97.62:42065 requested a full tablet report, sending...
I20260929 20:56:14.207525 14771 ts_manager.cc:194] Registered new tserver with Master: 07463aaa39ac4d4db7ff63a6657e7ee1 (127.0.97.2:43715)
I20260929 20:56:14.208652 14771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56843
W20260929 20:56:14.504767 15091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:14.505097 15091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:14.505209 15091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:14.512681 15091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:14.512792 15091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:14.512912 15091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:56:14.522972 15091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39597
--tserver_master_addrs=127.0.97.62:42065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:14.524305 15091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:14.525682 15091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:14.538237 15096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:14.538173 15097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:14.538419 15099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:14.539036 15091 server_base.cc:1034] running on GCE node
I20260929 20:56:14.539672 15091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:14.540560 15091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:14.541781 15091 hybrid_clock.cc:648] HybridClock initialized: now 1790715374541728 us; error 36 us; skew 500 ppm
I20260929 20:56:14.542088 15091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:14 dist-test-slave-m9g1 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:14.545018 15091 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:56:14.546407 15091 webserver.cc:460] Webserver started at http://127.0.97.3:39597/ using document root <none> and password file <none>
I20260929 20:56:14.547039 15091 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:14.547170 15091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:14.552349 15091 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:56:14.554987 15106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:14.556226 15091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:56:14.556362 15091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "d6e98474a4b0491684729581e8230622"
format_stamp: "Formatted at 2026-09-29 20:56:11 on dist-test-slave-m9g1"
server_key: "c19dc47e9f83adcd42bac4b9d781f463"
server_key_iv: "9a1659d989496bc30b03a3606d2d13d0"
server_key_version: "encryptionkey@0"
I20260929 20:56:14.557291 15091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:14.580569 15091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:14.584329 15091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:14.584568 15091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:14.585503 15091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:56:14.586859 15091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:56:14.586953 15091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:14.587052 15091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:56:14.587118 15091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:14.616410 15091 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38667
I20260929 20:56:14.616459 15218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38667 every 8 connection(s)
I20260929 20:56:14.617933 15091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:56:14.618346 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 15091
Sep 29 20:56:14 dist-test-slave-m9g1 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:56:14.634043 14958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:14.619801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53212 (local address 127.0.97.62:42065)
0929 20:56:14.619993 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:14.620000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:14.620190 (+ 190us) server_negotiation.cc:407] Connection header received
0929 20:56:14.620339 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:14.620345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:14.620429 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:14.620529 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:14.622394 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.623454 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.624784 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.625383 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.625501 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:14.628372 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:14.628378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:14.628383 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:14.628428 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:14.631502 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:14.632287 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:14.632292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:14.632295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:14.632437 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:14.632820 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:14.632824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:14.632828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:14.633152 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:14.633393 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:14.633662 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 20:56:14.633813 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:56:14.639559 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:14.621043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59261 (local address 127.0.97.62:42065)
0929 20:56:14.621420 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:14.621424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:14.622829 (+ 1405us) server_negotiation.cc:407] Connection header received
0929 20:56:14.624778 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:14.624783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:14.624843 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:14.624928 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:14.626722 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.627795 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.629796 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.630409 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.630526 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:14.634291 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:14.634297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:14.634302 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:14.634332 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:14.636705 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:14.637365 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:14.637368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:14.637371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:14.637472 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:14.637856 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:14.637860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:14.637864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:14.638103 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:14.638255 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:14.638957 (+ 702us) server_negotiation.cc:299] Negotiation successful
0929 20:56:14.639153 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":218,"threads_started":1}
I20260929 20:56:14.640489 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:14.621804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:59261)
0929 20:56:14.622658 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:14.622709 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:14.624521 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:14.625185 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:14.625196 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:14.625851 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:14.626546 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:14.626564 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.627928 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.627931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:14.629640 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:14.629655 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.630620 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.630627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:14.630842 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:14.631411 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:14.631430 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:14.634063 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:14.636835 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:14.636844 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:14.636860 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:14.637261 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:14.637569 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:14.637574 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:14.637578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:14.637754 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:14.638250 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:14.638258 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:14.638627 (+ 369us) client_negotiation.cc:769] Sending connection context
0929 20:56:14.638866 (+ 239us) client_negotiation.cc:240] Negotiation successful
0929 20:56:14.639400 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":343,"threads_started":1}
I20260929 20:56:14.641681 15219 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42065
I20260929 20:56:14.641978 15219 heartbeater.cc:464] Registering TS with master...
I20260929 20:56:14.642688 15219 heartbeater.cc:511] Master 127.0.97.62:42065 requested a full tablet report, sending...
I20260929 20:56:14.644032 14771 ts_manager.cc:194] Registered new tserver with Master: d6e98474a4b0491684729581e8230622 (127.0.97.3:38667)
I20260929 20:56:14.644806 14771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59261
I20260929 20:56:14.648571 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:56:14.655646 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:14.649509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53220 (local address 127.0.97.62:42065)
0929 20:56:14.649686 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:14.649691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:14.650054 (+ 363us) server_negotiation.cc:407] Connection header received
0929 20:56:14.650305 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:14.650310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:14.650465 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:14.650587 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:56:14.650994 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.651676 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.653231 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.653828 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.653955 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:14.654604 (+ 649us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:56:14.655101 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:56:14.655204 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:14.655328 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 20:56:14.655392 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:56:14.656555 14771 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:53220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:56:14.664007 14771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:56:14.686218 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:14.676241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.0.97.3:38667)
0929 20:56:14.677493 (+ 1252us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:14.677502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:14.678155 (+ 653us) server_negotiation.cc:407] Connection header received
0929 20:56:14.678295 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:14.678304 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:14.678581 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:14.678768 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:14.679827 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.681371 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.683910 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.685081 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.685262 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:14.685493 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:14.685725 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 20:56:14.685886 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1062,"thread_start_us":891,"threads_started":1}
I20260929 20:56:14.686311 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:14.675759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38667 (local address 127.0.0.1:33776)
0929 20:56:14.677907 (+ 2148us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:14.677925 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:14.678043 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:14.679104 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:14.679116 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:14.679157 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:14.679633 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:14.679646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.681503 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.681508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:14.683716 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:14.683734 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.685316 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.685322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:14.685498 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:14.685548 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 20:56:14.685663 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 20:56:14.685847 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2028,"mutex_wait_us":25,"spinlock_wait_cycles":18304,"thread_start_us":2325,"threads_started":1}
I20260929 20:56:14.686944 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:14.677116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43715 (local address 127.0.0.1:48556)
0929 20:56:14.679129 (+ 2013us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:14.679145 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:14.679260 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:14.680077 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:14.680081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:14.680106 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:14.680439 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:14.680451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.681990 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.681995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:14.683890 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:14.683909 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.686102 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.686108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:14.686325 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:14.686361 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:56:14.686443 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 20:56:14.686569 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1907,"mutex_wait_us":84,"thread_start_us":2105,"threads_started":1}
I20260929 20:56:14.687304 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:14.677742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48556 (local address 127.0.97.2:43715)
0929 20:56:14.678762 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:14.678772 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:14.679394 (+ 622us) server_negotiation.cc:407] Connection header received
0929 20:56:14.679482 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:14.679487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:14.679654 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:14.679820 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:14.680631 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.681862 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.684064 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.685624 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.685741 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:14.685953 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:14.686802 (+ 849us) server_negotiation.cc:299] Negotiation successful
0929 20:56:14.686964 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":719,"threads_started":1}
I20260929 20:56:14.687800 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:14.675615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44095 (local address 127.0.0.1:60352)
0929 20:56:14.676595 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:14.676637 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:14.676825 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:14.680699 (+ 3874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:14.680704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:14.680727 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:14.681098 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:14.681111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.683048 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.683054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:14.685199 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:14.685215 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.687046 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.687053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:14.687232 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:14.687264 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:56:14.687322 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 20:56:14.687436 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":35,"thread_start_us":543,"threads_started":1}
I20260929 20:56:14.689209 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:14.675794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60352 (local address 127.0.97.1:44095)
0929 20:56:14.680130 (+ 4336us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:14.680138 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:14.680162 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:56:14.680256 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:14.680263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:14.680482 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:14.681279 (+ 797us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:14.681453 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.682913 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.685389 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.686812 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.688300 (+ 1488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:14.688494 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:14.688723 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 20:56:14.688907 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4196,"thread_start_us":4093,"threads_started":1}
I20260929 20:56:14.690474 15154 tablet_service.cc:1467] Processing CreateTablet for tablet fc26f2bc2f9842f8b7634c7e53df610e (DEFAULT_TABLE table=test-table [id=d7eaa8f84067437cb6e4a8f0ad922f1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:56:14.690868 15023 tablet_service.cc:1467] Processing CreateTablet for tablet fc26f2bc2f9842f8b7634c7e53df610e (DEFAULT_TABLE table=test-table [id=d7eaa8f84067437cb6e4a8f0ad922f1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:56:14.691728 15154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc26f2bc2f9842f8b7634c7e53df610e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:14.692251 15023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc26f2bc2f9842f8b7634c7e53df610e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:14.692612 14891 tablet_service.cc:1467] Processing CreateTablet for tablet fc26f2bc2f9842f8b7634c7e53df610e (DEFAULT_TABLE table=test-table [id=d7eaa8f84067437cb6e4a8f0ad922f1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:56:14.693909 14891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc26f2bc2f9842f8b7634c7e53df610e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:14.703857 15235 tablet_bootstrap.cc:492] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622: Bootstrap starting.
I20260929 20:56:14.707415 15235 tablet_bootstrap.cc:654] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:14.707796 15236 tablet_bootstrap.cc:492] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1: Bootstrap starting.
I20260929 20:56:14.708948 15235 log.cc:824] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:14.709473 15237 tablet_bootstrap.cc:492] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19: Bootstrap starting.
I20260929 20:56:14.711176 15236 tablet_bootstrap.cc:654] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:14.712397 15236 log.cc:824] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:14.712603 15237 tablet_bootstrap.cc:654] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:14.712823 15235 tablet_bootstrap.cc:492] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622: No bootstrap required, opened a new log
I20260929 20:56:14.713148 15235 ts_tablet_manager.cc:1397] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260929 20:56:14.713723 15237 log.cc:824] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:14.715780 15236 tablet_bootstrap.cc:492] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1: No bootstrap required, opened a new log
I20260929 20:56:14.716111 15236 ts_tablet_manager.cc:1397] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 20:56:14.716530 15237 tablet_bootstrap.cc:492] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19: No bootstrap required, opened a new log
I20260929 20:56:14.716756 15237 ts_tablet_manager.cc:1397] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 20:56:14.723506 15237 raft_consensus.cc:348] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } }
I20260929 20:56:14.723546 15235 raft_consensus.cc:348] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } }
I20260929 20:56:14.723886 15237 raft_consensus.cc:374] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:14.723995 15237 raft_consensus.cc:729] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f226196eee47df894280b06f688e19, State: Initialized, Role: FOLLOWER
I20260929 20:56:14.724007 15235 raft_consensus.cc:374] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:14.724092 15235 raft_consensus.cc:729] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e98474a4b0491684729581e8230622, State: Initialized, Role: FOLLOWER
I20260929 20:56:14.724598 15237 consensus_queue.cc:260] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } }
I20260929 20:56:14.724790 15235 consensus_queue.cc:260] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } }
I20260929 20:56:14.725793 14956 heartbeater.cc:503] Master 127.0.97.62:42065 was elected leader, sending a full tablet report...
I20260929 20:56:14.726255 15236 raft_consensus.cc:348] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } }
I20260929 20:56:14.726680 15236 raft_consensus.cc:374] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:14.726773 15236 raft_consensus.cc:729] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07463aaa39ac4d4db7ff63a6657e7ee1, State: Initialized, Role: FOLLOWER
I20260929 20:56:14.727576 15236 consensus_queue.cc:260] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } }
I20260929 20:56:14.728295 15237 ts_tablet_manager.cc:1428] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260929 20:56:14.730019 15088 heartbeater.cc:503] Master 127.0.97.62:42065 was elected leader, sending a full tablet report...
I20260929 20:56:14.730511 15219 heartbeater.cc:503] Master 127.0.97.62:42065 was elected leader, sending a full tablet report...
I20260929 20:56:14.730684 15236 ts_tablet_manager.cc:1428] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260929 20:56:14.733155 15235 ts_tablet_manager.cc:1428] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622: Time spent starting tablet: real 0.020s user 0.011s sys 0.006s
W20260929 20:56:14.764025 14957 tablet.cc:2369] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:56:14.824136 15242 raft_consensus.cc:482] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:56:14.824419 15242 raft_consensus.cc:504] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } }
I20260929 20:56:14.826172 15242 leader_election.cc:288] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f226196eee47df894280b06f688e19 (127.0.97.1:44095), 07463aaa39ac4d4db7ff63a6657e7ee1 (127.0.97.2:43715)
I20260929 20:56:14.832100 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:14.826588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44095 (local address 127.0.97.3:42835)
0929 20:56:14.826789 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:14.826809 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:14.826960 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:14.827299 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:14.827304 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:14.827333 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:14.827510 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:14.827520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.828779 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.828784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:14.830376 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:14.830388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.831421 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.831427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:14.831585 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:14.831609 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:56:14.831685 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 20:56:14.831809 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 20:56:14.832096 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:14.826626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42835 (local address 127.0.97.1:44095)
0929 20:56:14.826833 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:14.826838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:14.826888 (+ 50us) server_negotiation.cc:407] Connection header received
0929 20:56:14.827106 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:14.827110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:14.827169 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:14.827249 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:14.827668 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.828615 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.830519 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.831273 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.831390 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:14.831481 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:14.831724 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 20:56:14.831856 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:56:14.832744 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:14.826731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51023 (local address 127.0.97.2:43715)
0929 20:56:14.826940 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:14.826946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:14.827566 (+ 620us) server_negotiation.cc:407] Connection header received
0929 20:56:14.827679 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:14.827691 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:14.827752 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:14.827818 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:14.828350 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.829207 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.831026 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.831830 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.831942 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:14.832037 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:14.832392 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 20:56:14.832526 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:56:14.832772 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:14.826587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43715 (local address 127.0.97.3:51023)
0929 20:56:14.827458 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:14.827472 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:14.827575 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:14.827851 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:14.827856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:14.827880 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:14.828211 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:14.828221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.829343 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.829348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:14.830873 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:14.830885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.832032 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.832038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:14.832189 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:14.832220 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:56:14.832349 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 20:56:14.832466 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":204,"thread_start_us":331,"threads_started":1}
I20260929 20:56:14.833245 14911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc26f2bc2f9842f8b7634c7e53df610e" candidate_uuid: "d6e98474a4b0491684729581e8230622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f226196eee47df894280b06f688e19" is_pre_election: true
I20260929 20:56:14.833847 14911 raft_consensus.cc:2454] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e98474a4b0491684729581e8230622 in term 0.
I20260929 20:56:14.833787 15043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc26f2bc2f9842f8b7634c7e53df610e" candidate_uuid: "d6e98474a4b0491684729581e8230622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" is_pre_election: true
I20260929 20:56:14.834297 15043 raft_consensus.cc:2454] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e98474a4b0491684729581e8230622 in term 0.
I20260929 20:56:14.834604 15110 leader_election.cc:302] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f226196eee47df894280b06f688e19, d6e98474a4b0491684729581e8230622; no voters:
I20260929 20:56:14.834960 15242 raft_consensus.cc:2781] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:56:14.835078 15242 raft_consensus.cc:482] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:56:14.835168 15242 raft_consensus.cc:3037] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:14.837286 15242 raft_consensus.cc:504] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } }
I20260929 20:56:14.837890 15242 leader_election.cc:288] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [CANDIDATE]: Term 1 election: Requested vote from peers 89f226196eee47df894280b06f688e19 (127.0.97.1:44095), 07463aaa39ac4d4db7ff63a6657e7ee1 (127.0.97.2:43715)
I20260929 20:56:14.838313 14911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc26f2bc2f9842f8b7634c7e53df610e" candidate_uuid: "d6e98474a4b0491684729581e8230622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f226196eee47df894280b06f688e19"
I20260929 20:56:14.838366 15043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc26f2bc2f9842f8b7634c7e53df610e" candidate_uuid: "d6e98474a4b0491684729581e8230622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1"
I20260929 20:56:14.838480 14911 raft_consensus.cc:3037] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:14.838552 15043 raft_consensus.cc:3037] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:14.840720 14911 raft_consensus.cc:2454] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e98474a4b0491684729581e8230622 in term 1.
I20260929 20:56:14.840873 15043 raft_consensus.cc:2454] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e98474a4b0491684729581e8230622 in term 1.
I20260929 20:56:14.841074 15110 leader_election.cc:302] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f226196eee47df894280b06f688e19, d6e98474a4b0491684729581e8230622; no voters:
I20260929 20:56:14.841351 15242 raft_consensus.cc:2781] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:56:14.841727 15242 raft_consensus.cc:686] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [term 1 LEADER]: Becoming Leader. State: Replica: d6e98474a4b0491684729581e8230622, State: Running, Role: LEADER
I20260929 20:56:14.842160 15242 consensus_queue.cc:237] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } }
I20260929 20:56:14.846989 14770 catalog_manager.cc:5495] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e98474a4b0491684729581e8230622 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d6e98474a4b0491684729581e8230622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6e98474a4b0491684729581e8230622" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 } health_report { overall_health: UNKNOWN } } }
W20260929 20:56:14.869885 15220 tablet.cc:2369] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:56:14.877707 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:14.872927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33778 (local address 127.0.97.3:38667)
0929 20:56:14.873088 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:14.873093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:14.873109 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:56:14.873259 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:14.873263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:14.873339 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:14.873460 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:56:14.873749 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.874494 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.875875 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.876261 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.876349 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:14.876784 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:56:14.877283 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:56:14.877391 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:14.877432 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 20:56:14.877497 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 20:56:14.885313 15043 raft_consensus.cc:1261] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1 [term 1 FOLLOWER]: Refusing update from remote peer d6e98474a4b0491684729581e8230622: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:56:14.885305 14911 raft_consensus.cc:1261] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19 [term 1 FOLLOWER]: Refusing update from remote peer d6e98474a4b0491684729581e8230622: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:56:14.886135 15242 consensus_queue.cc:1035] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f226196eee47df894280b06f688e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:56:14.886476 15245 consensus_queue.cc:1035] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07463aaa39ac4d4db7ff63a6657e7ee1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:56:14.896752 15247 mvcc.cc:205] Tried to move back new op lower bound from 7334770175520993280 to 7334770175357341696. Current Snapshot: MvccSnapshot[applied={T|T < 7334770175520993280}]
I20260929 20:56:14.910800 14770 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:53220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:56:14.911074 14770 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:53220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:56:14.914484 14770 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59: Sending DeleteTablet for 3 replicas of tablet fc26f2bc2f9842f8b7634c7e53df610e
I20260929 20:56:14.915634 15154 tablet_service.cc:1514] Processing DeleteTablet for tablet fc26f2bc2f9842f8b7634c7e53df610e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:56:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33776
I20260929 20:56:14.915632 14891 tablet_service.cc:1514] Processing DeleteTablet for tablet fc26f2bc2f9842f8b7634c7e53df610e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:56:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60352
I20260929 20:56:14.916349 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 14808
I20260929 20:56:14.916384 15023 tablet_service.cc:1514] Processing DeleteTablet for tablet fc26f2bc2f9842f8b7634c7e53df610e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:56:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48556
I20260929 20:56:14.918393 15257 tablet_replica.cc:330] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19: stopping tablet replica
I20260929 20:56:14.918923 15257 raft_consensus.cc:2229] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:56:14.919513 15257 raft_consensus.cc:2258] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:56:14.921804 15258 tablet_replica.cc:330] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1: stopping tablet replica
I20260929 20:56:14.922324 15258 raft_consensus.cc:2229] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:56:14.923887 15258 raft_consensus.cc:2258] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:56:14.924376 15256 tablet_replica.cc:330] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622: stopping tablet replica
I20260929 20:56:14.924819 15256 raft_consensus.cc:2229] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:56:14.925590 15256 raft_consensus.cc:2258] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:56:14.927933 15258 ts_tablet_manager.cc:1905] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:56:14.927933 15256 ts_tablet_manager.cc:1905] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:56:14.928054 15257 ts_tablet_manager.cc:1905] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:56:14.934000 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:14.917148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60364 (local address 127.0.97.1:44095)
0929 20:56:14.917417 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:14.917423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:14.917442 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:56:14.917599 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:14.917606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:14.917721 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:14.917822 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:14.919619 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.920834 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.922741 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:14.923167 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:14.923288 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:14.926510 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:14.926522 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:14.926539 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:14.926585 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:14.930376 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:14.931320 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:14.931330 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:14.931333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:14.931486 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:14.931974 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:14.931979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:14.931983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:14.932460 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:14.932767 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:14.933390 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 20:56:14.933620 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260929 20:56:14.934732 15256 ts_tablet_manager.cc:1918] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:56:14.934973 15256 log.cc:1196] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/fc26f2bc2f9842f8b7634c7e53df610e
I20260929 20:56:14.935334 15257 ts_tablet_manager.cc:1918] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:56:14.935456 14951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:14.935613 15257 log.cc:1196] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/fc26f2bc2f9842f8b7634c7e53df610e
I20260929 20:56:14.937021 15258 ts_tablet_manager.cc:1918] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:56:14.937170 15258 log.cc:1196] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/fc26f2bc2f9842f8b7634c7e53df610e
I20260929 20:56:14.937678 15258 ts_tablet_manager.cc:1939] T fc26f2bc2f9842f8b7634c7e53df610e P 07463aaa39ac4d4db7ff63a6657e7ee1: Deleting consensus metadata
I20260929 20:56:14.939510 14755 catalog_manager.cc:4841] TS 07463aaa39ac4d4db7ff63a6657e7ee1 (127.0.97.2:43715): tablet fc26f2bc2f9842f8b7634c7e53df610e (table test-table [id=d7eaa8f84067437cb6e4a8f0ad922f1d]) successfully deleted
I20260929 20:56:14.939819 15256 ts_tablet_manager.cc:1939] T fc26f2bc2f9842f8b7634c7e53df610e P d6e98474a4b0491684729581e8230622: Deleting consensus metadata
I20260929 20:56:14.941236 14757 catalog_manager.cc:4841] TS d6e98474a4b0491684729581e8230622 (127.0.97.3:38667): tablet fc26f2bc2f9842f8b7634c7e53df610e (table test-table [id=d7eaa8f84067437cb6e4a8f0ad922f1d]) successfully deleted
I20260929 20:56:15.052682 15257 ts_tablet_manager.cc:1939] T fc26f2bc2f9842f8b7634c7e53df610e P 89f226196eee47df894280b06f688e19: Deleting consensus metadata
I20260929 20:56:15.052873 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 14808
I20260929 20:56:15.068387 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 14960
W20260929 20:56:15.068606 14758 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44095 (error 108)
I20260929 20:56:15.071136 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:15.070644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44095 (local address 127.0.0.1:60370)
0929 20:56:15.070835 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:15.070954 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 20:56:15.071449 14758 catalog_manager.cc:4584] TS 89f226196eee47df894280b06f688e19 (127.0.97.1:44095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc26f2bc2f9842f8b7634c7e53df610e: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44095: connect: Connection refused (error 111)
I20260929 20:56:15.083052 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:15.069225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48564 (local address 127.0.97.2:43715)
0929 20:56:15.069406 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:15.069412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:15.069498 (+ 86us) server_negotiation.cc:407] Connection header received
0929 20:56:15.069718 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:15.069724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:15.069811 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:15.069908 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:15.071833 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:15.072597 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:15.073966 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:15.074432 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:15.074544 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:15.077275 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:15.077292 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:15.077308 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:15.077348 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:15.080357 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:15.081057 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:15.081065 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:15.081067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:15.081195 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:15.081545 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:15.081550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:15.081553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:15.081901 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:15.082172 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:15.082508 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 20:56:15.082724 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:56:15.083634 15083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:15.125181 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:15.124589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44095 (local address 127.0.0.1:60376)
0929 20:56:15.124843 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:15.124984 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 20:56:15.126427 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:15.125968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44095 (local address 127.0.0.1:60380)
0929 20:56:15.126136 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:15.126270 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 20:56:15.172279 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 14960
I20260929 20:56:15.185897 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 15091
I20260929 20:56:15.187057 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:15.186558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44095 (local address 127.0.0.1:60384)
0929 20:56:15.186767 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:15.186869 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 20:56:15.188479 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:15.188069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44095 (local address 127.0.0.1:60390)
0929 20:56:15.188239 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:15.188349 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 20:56:15.201486 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:15.186868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33786 (local address 127.0.97.3:38667)
0929 20:56:15.187035 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:15.187042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:15.187061 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:56:15.187130 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:15.187135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:15.187211 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:15.187361 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:15.188872 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:15.190071 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:15.191374 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:15.192043 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:15.192186 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:15.194617 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:15.194631 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:15.194651 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:15.194695 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:15.198623 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:15.199317 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:15.199331 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:15.199334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:15.199535 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:15.199853 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:15.199858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:15.199862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:15.200329 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:15.200646 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:15.200817 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 20:56:15.201048 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:56:15.202303 15214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:15.269367 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:15.268791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44095 (local address 127.0.0.1:60394)
0929 20:56:15.269035 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:15.269151 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 20:56:15.270478 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:15.270033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44095 (local address 127.0.0.1:60408)
0929 20:56:15.270259 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:15.270350 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 20:56:15.320262 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 15091
I20260929 20:56:15.336381 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 14738
I20260929 20:56:15.337594 14801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:15.450232 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 14738
I20260929 20:56:15.467700 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42065
--webserver_interface=127.0.97.62
--webserver_port=33889
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:56:15.787806 15264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:15.788105 15264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:15.788232 15264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:15.795480 15264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:56:15.795588 15264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:15.795658 15264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:15.795737 15264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:56:15.795794 15264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:56:15.805495 15264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:15.806910 15264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:15.808207 15264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:15.819249 15269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:15.819223 15272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:15.819178 15270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:15.820080 15264 server_base.cc:1034] running on GCE node
I20260929 20:56:15.820863 15264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:15.822463 15264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:15.823679 15264 hybrid_clock.cc:648] HybridClock initialized: now 1790715375823620 us; error 54 us; skew 500 ppm
I20260929 20:56:15.824033 15264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:15 dist-test-slave-m9g1 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:15.843890 15264 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:56:15.845490 15264 webserver.cc:460] Webserver started at http://127.0.97.62:33889/ using document root <none> and password file <none>
I20260929 20:56:15.846233 15264 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:15.846366 15264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:15.851840 15264 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:56:15.854626 15279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:15.855955 15264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:56:15.856114 15264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "d7ce0a71a1d343d6b3656da6ac4baf59"
format_stamp: "Formatted at 2026-09-29 20:56:10 on dist-test-slave-m9g1"
server_key: "0a4f7f99467529bdfe7ce10370610136"
server_key_iv: "7f3dace5656679fd14a5f3bae5d7f1d1"
server_key_version: "encryptionkey@0"
I20260929 20:56:15.857059 15264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:15.879021 15264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:15.883061 15264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:15.883318 15264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:15.900257 15264 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42065
I20260929 20:56:15.900285 15330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42065 every 8 connection(s)
I20260929 20:56:15.901706 15264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:56:15.908282 15331 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:56:15.909950 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 15264
I20260929 20:56:15.910748 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44095
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42065
--webserver_port=46451
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:56:15.911734 15331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59: Bootstrap starting.
I20260929 20:56:15.931627 15331 log.cc:824] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:15.947782 15331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:56:15.948294 15331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59: Bootstrap complete.
I20260929 20:56:15.957248 15331 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42065 } }
I20260929 20:56:15.958149 15331 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ce0a71a1d343d6b3656da6ac4baf59, State: Initialized, Role: FOLLOWER
I20260929 20:56:15.958773 15331 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42065 } }
I20260929 20:56:15.959002 15331 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:56:15.959111 15331 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:56:15.959268 15331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 2 FOLLOWER]: Advancing to term 3
I20260929 20:56:15.963091 15331 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42065 } }
I20260929 20:56:15.963557 15331 leader_election.cc:302] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7ce0a71a1d343d6b3656da6ac4baf59; no voters:
I20260929 20:56:15.964195 15331 leader_election.cc:288] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 20:56:15.964329 15335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 20:56:15.966414 15331 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:56:15.970474 15335 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [term 3 LEADER]: Becoming Leader. State: Replica: d7ce0a71a1d343d6b3656da6ac4baf59, State: Running, Role: LEADER
I20260929 20:56:15.971163 15335 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42065 } }
I20260929 20:56:15.973791 15336 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42065 } } }
I20260929 20:56:15.974179 15336 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [sys.catalog]: This master's current role is: LEADER
I20260929 20:56:15.974785 15337 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7ce0a71a1d343d6b3656da6ac4baf59. Latest consensus state: current_term: 3 leader_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ce0a71a1d343d6b3656da6ac4baf59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42065 } } }
I20260929 20:56:15.975054 15337 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ce0a71a1d343d6b3656da6ac4baf59 [sys.catalog]: This master's current role is: LEADER
I20260929 20:56:15.988081 15343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:56:15.997081 15343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:56:15.998771 15343 catalog_manager.cc:1259] Loaded cluster ID: 435fd7e57fd946648322b67d3af39e63
I20260929 20:56:15.999238 15343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 20:56:16.001427 15343 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f257d7d99ad google::LogMessage::Fail() at ??:0
@ 0x7f257d7dc88a google::LogMessage::SendToLog() at ??:0
@ 0x7f257d7d94de google::LogMessage::Flush() at ??:0
@ 0x7f257d7daa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f259320c512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f2593209976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2593291610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f2593291439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2580425345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f258042891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f2580428799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2580403eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f25836dd6db start_thread at ??:0
@ 0x7f257c67361f clone at ??:0
W20260929 20:56:16.285441 15333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:16.285742 15333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:16.285880 15333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:16.293298 15333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:16.293411 15333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:16.293560 15333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:56:16.303628 15333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=46451
--tserver_master_addrs=127.0.97.62:42065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:16.304946 15333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:16.306308 15333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:16.319171 15372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:16.319118 15373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:16.319175 15375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:16.320050 15333 server_base.cc:1034] running on GCE node
I20260929 20:56:16.320693 15333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:16.321781 15333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:16.323014 15333 hybrid_clock.cc:648] HybridClock initialized: now 1790715376322955 us; error 50 us; skew 500 ppm
I20260929 20:56:16.323366 15333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:16 dist-test-slave-m9g1 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:16.326748 15333 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:56:16.328203 15333 webserver.cc:460] Webserver started at http://127.0.97.1:46451/ using document root <none> and password file <none>
I20260929 20:56:16.328924 15333 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:16.329028 15333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:16.334569 15333 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:56:16.337384 15382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:16.338902 15333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:56:16.339062 15333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "89f226196eee47df894280b06f688e19"
format_stamp: "Formatted at 2026-09-29 20:56:10 on dist-test-slave-m9g1"
server_key: "0d423f27789b6d0c53b28cff79169ee8"
server_key_iv: "f7c9812520a5f6be89dd4e5ae30c2c26"
server_key_version: "encryptionkey@0"
I20260929 20:56:16.340070 15333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:16.360587 15333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:16.365103 15333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:16.365401 15333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:16.366516 15333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:56:16.367870 15333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:56:16.367969 15333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:16.368069 15333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:56:16.368137 15333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:16.397269 15333 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44095
I20260929 20:56:16.397338 15494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44095 every 8 connection(s)
I20260929 20:56:16.398797 15333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:56:16.403192 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 15333
I20260929 20:56:16.403807 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:43715
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42065
--webserver_port=38513
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:56:16.403986 15497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:16.401895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:39161)
0929 20:56:16.402727 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:16.402933 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":327,"threads_started":1}
W20260929 20:56:16.404666 15495 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42065 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42065: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
I20260929 20:56:16.405539 15497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:16.405207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:50025)
0929 20:56:16.405362 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:16.405431 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 20:56:16.406379 15497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:16.406030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:40023)
0929 20:56:16.406162 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:16.406267 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 20:56:16.406589 15495 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:56:16.724138 15498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:16.724431 15498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:16.724556 15498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:16.731797 15498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:16.731915 15498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:16.732066 15498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:56:16.741905 15498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:43715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=38513
--tserver_master_addrs=127.0.97.62:42065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:16.743268 15498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:16.744643 15498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:16.757257 15506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:16.757315 15504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:16.757280 15503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:16.757721 15498 server_base.cc:1034] running on GCE node
I20260929 20:56:16.758504 15498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:16.759406 15498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:16.760598 15498 hybrid_clock.cc:648] HybridClock initialized: now 1790715376760541 us; error 48 us; skew 500 ppm
I20260929 20:56:16.760905 15498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:16 dist-test-slave-m9g1 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:16.763917 15498 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:56:16.765396 15498 webserver.cc:460] Webserver started at http://127.0.97.2:38513/ using document root <none> and password file <none>
I20260929 20:56:16.766067 15498 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:16.766208 15498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:16.771463 15498 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:56:16.776201 15513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:16.777508 15498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:56:16.777661 15498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "07463aaa39ac4d4db7ff63a6657e7ee1"
format_stamp: "Formatted at 2026-09-29 20:56:11 on dist-test-slave-m9g1"
server_key: "855385f3890720b51872262db871a6a4"
server_key_iv: "4c5f85ece3f3cc271b83700c60805dfb"
server_key_version: "encryptionkey@0"
I20260929 20:56:16.778668 15498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:16.794833 15498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:16.798826 15498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:16.799088 15498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:16.800081 15498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:56:16.801383 15498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:56:16.801476 15498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:16.801580 15498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:56:16.801651 15498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:16.831314 15498 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43715
I20260929 20:56:16.831377 15625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43715 every 8 connection(s)
I20260929 20:56:16.832712 15498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:56:16.834656 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 15498
I20260929 20:56:16.835341 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38667
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42065
--webserver_port=39597
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:56:16.837795 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:16.835788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:51133)
0929 20:56:16.836605 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:16.836871 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":316,"threads_started":1}
W20260929 20:56:16.838539 15626 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42065 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42065: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
I20260929 20:56:16.839453 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:16.839079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:54487)
0929 20:56:16.839245 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:16.839324 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 20:56:16.840365 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:16.840009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:49643)
0929 20:56:16.840151 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:16.840239 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 20:56:16.840612 15626 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:56:17.149035 15629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:17.149302 15629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:17.149425 15629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:17.156533 15629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:17.156637 15629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:17.156793 15629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:56:17.166267 15629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39597
--tserver_master_addrs=127.0.97.62:42065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:17.167558 15629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:17.168866 15629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:17.181181 15635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:17.181136 15637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:17.181108 15634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:17.181980 15629 server_base.cc:1034] running on GCE node
I20260929 20:56:17.182626 15629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:17.183580 15629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:17.184769 15629 hybrid_clock.cc:648] HybridClock initialized: now 1790715377184719 us; error 45 us; skew 500 ppm
I20260929 20:56:17.185081 15629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:17 dist-test-slave-m9g1 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:17.187950 15629 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:56:17.189319 15629 webserver.cc:460] Webserver started at http://127.0.97.3:39597/ using document root <none> and password file <none>
I20260929 20:56:17.189939 15629 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:17.190058 15629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:17.195165 15629 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:56:17.197656 15644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:17.198923 15629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:56:17.199077 15629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "d6e98474a4b0491684729581e8230622"
format_stamp: "Formatted at 2026-09-29 20:56:11 on dist-test-slave-m9g1"
server_key: "c19dc47e9f83adcd42bac4b9d781f463"
server_key_iv: "9a1659d989496bc30b03a3606d2d13d0"
server_key_version: "encryptionkey@0"
I20260929 20:56:17.200002 15629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:17.215638 15629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:17.219475 15629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:17.219729 15629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:17.220670 15629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:56:17.221979 15629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:56:17.222072 15629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:17.222177 15629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:56:17.222265 15629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 20:56:17.252221 15629 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38667
I20260929 20:56:17.252302 15756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38667 every 8 connection(s)
I20260929 20:56:17.253684 15629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:56:17.256438 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 15629
I20260929 20:56:17.258714 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:17.256674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:38827)
0929 20:56:17.257398 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:17.257612 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":287,"threads_started":1}
W20260929 20:56:17.259334 15757 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42065 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42065: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
I20260929 20:56:17.260083 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:17.259772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:49537)
0929 20:56:17.259921 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:17.259985 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 20:56:17.260766 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:17.260489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:55469)
0929 20:56:17.260600 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:17.260668 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 20:56:17.260977 15757 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 20:56:17.407785 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:17.407052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:48921)
0929 20:56:17.407488 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:17.407573 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":202,"threads_started":1}
I20260929 20:56:17.841975 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:17.841157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:53921)
0929 20:56:17.841615 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:17.841705 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":247,"threads_started":1}
I20260929 20:56:18.262531 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:18.261604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:33687)
0929 20:56:18.262114 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:18.262251 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":247,"threads_started":1}
I20260929 20:56:18.409543 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:18.408746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:33625)
0929 20:56:18.409276 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:18.409363 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":273,"threads_started":1}
I20260929 20:56:18.843734 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:18.842951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:55145)
0929 20:56:18.843441 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:18.843558 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":215,"threads_started":1}
I20260929 20:56:19.264173 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:19.263413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:40633)
0929 20:56:19.263884 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:19.263972 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":242,"threads_started":1}
I20260929 20:56:19.411218 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:19.410477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:45407)
0929 20:56:19.410928 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:19.411007 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":246,"threads_started":1}
I20260929 20:56:19.845463 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:19.844723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:33947)
0929 20:56:19.845198 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:19.845286 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":243,"threads_started":1}
I20260929 20:56:20.265611 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:20.264928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:33287)
0929 20:56:20.265346 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:20.265427 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":216,"threads_started":1}
I20260929 20:56:20.412902 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:20.412145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:43183)
0929 20:56:20.412609 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:20.412699 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":239,"threads_started":1}
I20260929 20:56:20.847210 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:20.846451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:52473)
0929 20:56:20.846922 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:20.847008 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":236,"threads_started":1}
I20260929 20:56:21.267481 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:21.266640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:32829)
0929 20:56:21.267114 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:21.267234 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":234,"threads_started":1}
I20260929 20:56:21.414625 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:21.413795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:40305)
0929 20:56:21.414362 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:21.414447 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":255,"threads_started":1}
I20260929 20:56:21.848958 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:21.848215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:54301)
0929 20:56:21.848704 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:21.848785 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":239,"threads_started":1}
I20260929 20:56:22.269081 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:22.268370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:39831)
0929 20:56:22.268838 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:22.268918 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":232,"threads_started":1}
I20260929 20:56:22.416342 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:22.415569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:58929)
0929 20:56:22.416055 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:22.416169 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":246,"threads_started":1}
I20260929 20:56:22.850742 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:22.849911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:39249)
0929 20:56:22.850460 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:22.850566 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":277,"threads_started":1}
I20260929 20:56:23.270947 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:23.270014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:52543)
0929 20:56:23.270696 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:23.270782 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":464,"threads_started":1}
I20260929 20:56:23.418087 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:23.417217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:33563)
0929 20:56:23.417786 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:23.417872 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":235,"threads_started":1}
I20260929 20:56:23.852576 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:23.851703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:47973)
0929 20:56:23.852260 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:23.852341 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":281,"threads_started":1}
I20260929 20:56:24.272666 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:24.271835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:40005)
0929 20:56:24.272363 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:24.272449 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":234,"threads_started":1}
I20260929 20:56:24.419879 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:24.419122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:45165)
0929 20:56:24.419609 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:24.419697 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":257,"threads_started":1}
I20260929 20:56:24.854274 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:24.853496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:39719)
0929 20:56:24.853969 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:24.854064 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":227,"threads_started":1}
I20260929 20:56:25.274518 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:25.273692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:39337)
0929 20:56:25.274236 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:25.274324 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":285,"threads_started":1}
I20260929 20:56:25.421548 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:25.420770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:56953)
0929 20:56:25.421269 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:25.421362 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":254,"threads_started":1}
I20260929 20:56:25.855902 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:25.855151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:58389)
0929 20:56:25.855636 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:25.855724 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":239,"threads_started":1}
I20260929 20:56:26.276234 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:26.275471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:49753)
0929 20:56:26.275962 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:26.276053 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":235,"threads_started":1}
I20260929 20:56:26.423408 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:26.422584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:36037)
0929 20:56:26.423127 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:26.423239 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":256,"threads_started":1}
I20260929 20:56:26.857697 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:26.856887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:58683)
0929 20:56:26.857397 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:26.857490 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":261,"threads_started":1}
I20260929 20:56:27.278096 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:27.277280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:53969)
0929 20:56:27.277831 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:27.277917 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":319,"threads_started":1}
I20260929 20:56:27.425305 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:27.424468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:59795)
0929 20:56:27.425010 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:27.425118 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":281,"threads_started":1}
I20260929 20:56:27.859411 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:27.858702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:50935)
0929 20:56:27.859134 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:27.859229 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":207,"threads_started":1}
I20260929 20:56:28.279831 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:28.279109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:37281)
0929 20:56:28.279546 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:28.279646 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":208,"threads_started":1}
I20260929 20:56:28.427240 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:28.426448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:45603)
0929 20:56:28.426969 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:28.427067 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":278,"threads_started":1}
I20260929 20:56:28.861042 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:28.860327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:44981)
0929 20:56:28.860797 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:28.860880 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":253,"threads_started":1}
I20260929 20:56:29.281554 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:29.280833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:60819)
0929 20:56:29.281312 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:29.281398 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":243,"threads_started":1}
I20260929 20:56:29.429190 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:29.428370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:40557)
0929 20:56:29.428895 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:29.428995 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":240,"threads_started":1}
I20260929 20:56:29.863008 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:29.862091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:54303)
0929 20:56:29.862666 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:29.862787 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":257,"threads_started":1}
I20260929 20:56:30.283375 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:30.282565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:49875)
0929 20:56:30.283099 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:30.283186 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":280,"threads_started":1}
I20260929 20:56:30.430938 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:30.430223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:43003)
0929 20:56:30.430661 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:30.430755 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":212,"threads_started":1}
I20260929 20:56:30.864750 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:30.864010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:50351)
0929 20:56:30.864454 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:30.864544 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":226,"threads_started":1}
I20260929 20:56:31.285380 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:31.284318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:43157)
0929 20:56:31.285042 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:31.285167 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":304,"threads_started":1}
I20260929 20:56:31.432752 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:31.432032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.1:57955)
0929 20:56:31.432486 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:31.432569 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":242,"threads_started":1}
I20260929 20:56:31.866665 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:31.865926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.2:48873)
0929 20:56:31.866400 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:31.866481 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":214,"threads_started":1}
I20260929 20:56:32.264775 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 15333
I20260929 20:56:32.282483 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:32.266004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52934 (local address 127.0.97.1:44095)
0929 20:56:32.266597 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:32.266616 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:32.266660 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:56:32.268733 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:32.268743 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:32.269159 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:32.269441 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:32.270783 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:32.272020 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:32.273256 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:32.273921 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:32.274052 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:32.276478 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:32.276486 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:32.276501 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:32.276537 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:32.279204 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:32.279990 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:32.279998 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:32.280005 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:32.280184 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:32.280565 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:32.280569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:32.280571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:32.281102 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:32.281347 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:32.281698 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 20:56:32.282141 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":289,"threads_started":1}
I20260929 20:56:32.283914 15490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:32.299425 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:32.286543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42065 (local address 127.0.97.3:35941)
0929 20:56:32.299050 (+ 12507us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:32.299166 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12354,"thread_start_us":12231,"threads_started":1}
I20260929 20:56:32.365832 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 15333
I20260929 20:56:32.379290 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 15498
I20260929 20:56:32.396090 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:32.380256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41684 (local address 127.0.97.2:43715)
0929 20:56:32.380763 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:32.380778 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:32.380824 (+ 46us) server_negotiation.cc:407] Connection header received
0929 20:56:32.382471 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:32.382481 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:32.382937 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:32.383232 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:32.384684 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:32.385966 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:32.387308 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:32.387981 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:32.388110 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:32.390552 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:32.390560 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:32.390575 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:32.390616 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:32.393277 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:32.393886 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:32.393892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:32.393899 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:32.394051 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:32.394373 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:32.394376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:32.394379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:32.394868 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:32.395086 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:32.395399 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 20:56:32.395786 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":225,"threads_started":1}
I20260929 20:56:32.397373 15621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:32.480070 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 15498
I20260929 20:56:32.494064 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 15629
I20260929 20:56:32.511893 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:32.495009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.0.97.3:38667)
0929 20:56:32.495565 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:32.495581 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:32.495622 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:56:32.497231 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:32.497240 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:32.497686 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:32.498031 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:32.499615 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:32.500855 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:32.502209 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:32.502968 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:32.503163 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:32.505817 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:32.505825 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:32.505842 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:32.505900 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:32.508622 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:32.509315 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:32.509323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:32.509329 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:32.509500 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:32.509864 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:32.509868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:32.509877 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:32.510453 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:32.510745 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:32.511180 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 20:56:32.511589 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":235,"threads_started":1}
I20260929 20:56:32.513417 15752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:32.599601 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 15629
2026-09-29T20:56:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23847 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:56:32 dist-test-slave-m9g1 krb5kdc[15814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:56:32 dist-test-slave-m9g1 krb5kdc[15814](info): set up 2 sockets
Sep 29 20:56:32 dist-test-slave-m9g1 krb5kdc[15814](info): commencing operation
krb5kdc: starting...
W20260929 20:56:33.606429 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:56:33 dist-test-slave-m9g1 krb5kdc[15814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:56:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:56:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:56:33.833671 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37141
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:56:34.154547 15827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:34.154843 15827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:34.154942 15827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:34.162060 15827 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 20:56:34.162171 15827 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 20:56:34.162240 15827 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 20:56:34.162312 15827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:56:34.162375 15827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:34.162442 15827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:34.162485 15827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:56:34.162535 15827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:56:34.172559 15827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:34.173815 15827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:34.175119 15827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:34.186357 15832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:34.186528 15833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:34.186558 15835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:34.187216 15827 server_base.cc:1034] running on GCE node
I20260929 20:56:34.187987 15827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:34.189383 15827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:34.190593 15827 hybrid_clock.cc:648] HybridClock initialized: now 1790715394190544 us; error 43 us; skew 500 ppm
I20260929 20:56:34.190910 15827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:34 dist-test-slave-m9g1 krb5kdc[15814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:34.193836 15827 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:56:34.195327 15827 webserver.cc:460] Webserver started at http://127.0.97.62:38113/ using document root <none> and password file <none>
I20260929 20:56:34.196017 15827 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:34.196113 15827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:34.196417 15827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:56:34.199385 15827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "f4707c4c1d364f9aa579a7f8c75be245"
format_stamp: "Formatted at 2026-09-29 20:56:34 on dist-test-slave-m9g1"
server_key: "f978a6b74d0b582be12154a001318628"
server_key_iv: "68e7bb4f2566df506303fc4036c367b2"
server_key_version: "encryptionkey@0"
I20260929 20:56:34.200119 15827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "f4707c4c1d364f9aa579a7f8c75be245"
format_stamp: "Formatted at 2026-09-29 20:56:34 on dist-test-slave-m9g1"
server_key: "f978a6b74d0b582be12154a001318628"
server_key_iv: "68e7bb4f2566df506303fc4036c367b2"
server_key_version: "encryptionkey@0"
I20260929 20:56:34.205180 15827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:56:34.208562 15842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:34.209901 15827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:56:34.210068 15827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "f4707c4c1d364f9aa579a7f8c75be245"
format_stamp: "Formatted at 2026-09-29 20:56:34 on dist-test-slave-m9g1"
server_key: "f978a6b74d0b582be12154a001318628"
server_key_iv: "68e7bb4f2566df506303fc4036c367b2"
server_key_version: "encryptionkey@0"
I20260929 20:56:34.210305 15827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:34.229465 15827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:34.233592 15827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:34.233844 15827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:34.251507 15827 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37141
I20260929 20:56:34.251545 15893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37141 every 8 connection(s)
I20260929 20:56:34.252925 15827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:56:34.256739 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 15827
I20260929 20:56:34.256963 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:56:34.257058 15894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:34.257817 388 external_mini_cluster.cc:1428] Setting key d3528c9d67217201cb0b7e8a2b1bac02
I20260929 20:56:34.271409 15894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245: Bootstrap starting.
Sep 29 20:56:34 dist-test-slave-m9g1 krb5kdc[15814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:56:34.274647 15894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:34.275755 15894 log.cc:824] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:34.278543 15894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245: No bootstrap required, opened a new log
I20260929 20:56:34.279924 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:34.259910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40256 (local address 127.0.97.62:37141)
0929 20:56:34.260763 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:34.260775 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:34.260818 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:56:34.262423 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:34.262449 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:34.262914 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:34.263279 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:34.264972 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:34.266275 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:34.268156 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:34.268965 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:34.269117 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:34.273198 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:34.273207 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:34.273221 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:34.273258 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:34.276214 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:34.276836 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:34.276843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:34.276850 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:34.277005 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:34.277353 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:34.277357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:34.277359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:34.277891 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:34.278163 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:34.278610 (+ 447us) server_negotiation.cc:299] Negotiation successful
0929 20:56:34.278995 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":379,"threads_started":1}
I20260929 20:56:34.286240 15894 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4707c4c1d364f9aa579a7f8c75be245" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } }
I20260929 20:56:34.286628 15894 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:34.286716 15894 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4707c4c1d364f9aa579a7f8c75be245, State: Initialized, Role: FOLLOWER
I20260929 20:56:34.287278 15894 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4707c4c1d364f9aa579a7f8c75be245" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } }
I20260929 20:56:34.287494 15894 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:56:34.287624 15894 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:56:34.287784 15894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:34.290274 15894 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4707c4c1d364f9aa579a7f8c75be245" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } }
I20260929 20:56:34.290767 15894 leader_election.cc:302] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4707c4c1d364f9aa579a7f8c75be245; no voters:
I20260929 20:56:34.291304 15894 leader_election.cc:288] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:56:34.291399 15899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:56:34.292053 15899 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 1 LEADER]: Becoming Leader. State: Replica: f4707c4c1d364f9aa579a7f8c75be245, State: Running, Role: LEADER
I20260929 20:56:34.292584 15899 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4707c4c1d364f9aa579a7f8c75be245" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } }
I20260929 20:56:34.293320 15894 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:56:34.295204 15901 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4707c4c1d364f9aa579a7f8c75be245. Latest consensus state: current_term: 1 leader_uuid: "f4707c4c1d364f9aa579a7f8c75be245" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4707c4c1d364f9aa579a7f8c75be245" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } }
I20260929 20:56:34.296252 15901 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [sys.catalog]: This master's current role is: LEADER
I20260929 20:56:34.295171 15900 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4707c4c1d364f9aa579a7f8c75be245" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4707c4c1d364f9aa579a7f8c75be245" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } }
I20260929 20:56:34.297160 15900 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [sys.catalog]: This master's current role is: LEADER
I20260929 20:56:34.297966 15905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:56:34.303498 15905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:56:34.312681 15905 catalog_manager.cc:1347] Generated new cluster ID: ae20801ac44f41adaaef378b12656ebe
I20260929 20:56:34.312798 15905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:56:34.327487 15905 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:56:34.329397 15905 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:56:34.342789 15905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245: Generated new TSK 0
I20260929 20:56:34.343974 15905 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:56:34.401268 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37141
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:56:34.735215 15920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:34.735656 15920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:34.735850 15920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:34.746448 15920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:34.746660 15920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:34.746905 15920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:56:34.762176 15920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:34.764144 15920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:34.766072 15920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:34.781642 15928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:34.781658 15926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:34.781860 15925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:34.782491 15920 server_base.cc:1034] running on GCE node
I20260929 20:56:34.783077 15920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:34.783981 15920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:34.785177 15920 hybrid_clock.cc:648] HybridClock initialized: now 1790715394785125 us; error 47 us; skew 500 ppm
I20260929 20:56:34.785463 15920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:34 dist-test-slave-m9g1 krb5kdc[15814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:34.795590 15920 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:56:34.797122 15920 webserver.cc:460] Webserver started at http://127.0.97.1:43945/ using document root <none> and password file <none>
I20260929 20:56:34.797834 15920 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:34.797931 15920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:34.798230 15920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:56:34.801074 15920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "088f9fb429d34b12b20ab1e6615787cb"
format_stamp: "Formatted at 2026-09-29 20:56:34 on dist-test-slave-m9g1"
server_key: "fc00edefbfa8aaa12f6e0b9eff507e7c"
server_key_iv: "4f41073b8b11f95d76740283b051a87c"
server_key_version: "encryptionkey@0"
I20260929 20:56:34.801753 15920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "088f9fb429d34b12b20ab1e6615787cb"
format_stamp: "Formatted at 2026-09-29 20:56:34 on dist-test-slave-m9g1"
server_key: "fc00edefbfa8aaa12f6e0b9eff507e7c"
server_key_iv: "4f41073b8b11f95d76740283b051a87c"
server_key_version: "encryptionkey@0"
I20260929 20:56:34.806648 15920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 20:56:34.809801 15935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:34.811172 15920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:56:34.811304 15920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "088f9fb429d34b12b20ab1e6615787cb"
format_stamp: "Formatted at 2026-09-29 20:56:34 on dist-test-slave-m9g1"
server_key: "fc00edefbfa8aaa12f6e0b9eff507e7c"
server_key_iv: "4f41073b8b11f95d76740283b051a87c"
server_key_version: "encryptionkey@0"
I20260929 20:56:34.811466 15920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:34.853053 15920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:34.857177 15920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:34.857417 15920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:34.858327 15920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:56:34.859609 15920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:56:34.859678 15920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:34.859766 15920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:56:34.859807 15920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:34.889328 15920 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45865
I20260929 20:56:34.889369 16047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45865 every 8 connection(s)
I20260929 20:56:34.890621 15920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:56:34.896915 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 15920
I20260929 20:56:34.897128 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:56:34.897914 388 external_mini_cluster.cc:1428] Setting key d62ac7c59582808b054421b4d57a5456
Sep 29 20:56:34 dist-test-slave-m9g1 krb5kdc[15814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:56:34.912437 16050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:34.892990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48691 (local address 127.0.97.62:37141)
0929 20:56:34.893443 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:34.893449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:34.894418 (+ 969us) server_negotiation.cc:407] Connection header received
0929 20:56:34.896316 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:34.896322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:34.896418 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:34.896523 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:34.898658 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:34.899677 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:34.902059 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:34.902517 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:34.902630 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:34.906677 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:34.906682 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:34.906686 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:34.906720 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:34.909427 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:34.910220 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:34.910226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:34.910230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:34.910374 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:34.910757 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:34.910761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:34.910764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:34.911079 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:34.911220 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:34.911870 (+ 650us) server_negotiation.cc:299] Negotiation successful
0929 20:56:34.912090 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":208,"threads_started":1}
I20260929 20:56:34.913285 16051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:34.893500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:48691)
0929 20:56:34.894263 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:34.894307 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:34.895997 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:34.896813 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:34.896825 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:34.897586 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:34.898464 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:34.898493 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:34.899930 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:34.899934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:34.901887 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:34.901906 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:34.902746 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:34.902753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:34.903040 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:34.903743 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:34.903773 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:34.906430 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:34.909574 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:34.909581 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:34.909596 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:34.910052 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:34.910473 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:34.910476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:34.910479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:34.910663 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:34.911194 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:34.911199 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:34.911515 (+ 316us) client_negotiation.cc:769] Sending connection context
0929 20:56:34.911741 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 20:56:34.912152 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":272,"threads_started":1}
I20260929 20:56:34.914757 16048 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37141
I20260929 20:56:34.915061 16048 heartbeater.cc:464] Registering TS with master...
I20260929 20:56:34.915773 16048 heartbeater.cc:511] Master 127.0.97.62:37141 requested a full tablet report, sending...
I20260929 20:56:34.917817 15859 ts_manager.cc:194] Registered new tserver with Master: 088f9fb429d34b12b20ab1e6615787cb (127.0.97.1:45865)
I20260929 20:56:34.919272 15859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48691
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:56:34.949841 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37141
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:56:35.262434 16054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:35.262693 16054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:35.262809 16054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:35.270298 16054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:35.270409 16054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:35.270555 16054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:56:35.280385 16054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:35.281668 16054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:35.282992 16054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:35.295773 16062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:35.295835 16059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:35.295861 16060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:35.296900 16054 server_base.cc:1034] running on GCE node
I20260929 20:56:35.297483 16054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:35.298480 16054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:35.299691 16054 hybrid_clock.cc:648] HybridClock initialized: now 1790715395299635 us; error 57 us; skew 500 ppm
I20260929 20:56:35.300012 16054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:35 dist-test-slave-m9g1 krb5kdc[15814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:35.303117 16054 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:56:35.304517 16054 webserver.cc:460] Webserver started at http://127.0.97.2:33685/ using document root <none> and password file <none>
I20260929 20:56:35.305208 16054 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:35.305321 16054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:35.305616 16054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:56:35.308509 16054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de"
format_stamp: "Formatted at 2026-09-29 20:56:35 on dist-test-slave-m9g1"
server_key: "4d7c59596e5058aaa4586c868e60b4dc"
server_key_iv: "83e0f028a35d9ed4125921ea6779368a"
server_key_version: "encryptionkey@0"
I20260929 20:56:35.309266 16054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de"
format_stamp: "Formatted at 2026-09-29 20:56:35 on dist-test-slave-m9g1"
server_key: "4d7c59596e5058aaa4586c868e60b4dc"
server_key_iv: "83e0f028a35d9ed4125921ea6779368a"
server_key_version: "encryptionkey@0"
I20260929 20:56:35.314298 16054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260929 20:56:35.317719 16069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:35.319082 16054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:56:35.319211 16054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de"
format_stamp: "Formatted at 2026-09-29 20:56:35 on dist-test-slave-m9g1"
server_key: "4d7c59596e5058aaa4586c868e60b4dc"
server_key_iv: "83e0f028a35d9ed4125921ea6779368a"
server_key_version: "encryptionkey@0"
I20260929 20:56:35.319372 16054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:35.336721 16054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:35.340891 16054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:35.341126 16054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:35.342020 16054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:56:35.343374 16054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:56:35.343448 16054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:35.343537 16054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:56:35.343600 16054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:35.372422 16054 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33027
I20260929 20:56:35.372488 16181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33027 every 8 connection(s)
I20260929 20:56:35.373677 16054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:56:35.380841 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 16054
I20260929 20:56:35.381060 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:56:35.381515 388 external_mini_cluster.cc:1428] Setting key 67567373447a72808e7246aca44a9ef6
Sep 29 20:56:35 dist-test-slave-m9g1 krb5kdc[15814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:56:35.393736 16050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:35.376172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39691 (local address 127.0.97.62:37141)
0929 20:56:35.376354 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:35.376359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:35.377538 (+ 1179us) server_negotiation.cc:407] Connection header received
0929 20:56:35.379407 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:35.379411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:35.379484 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:35.379578 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:35.381077 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.381840 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.383601 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.384041 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.384140 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:35.388224 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:35.388229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:35.388233 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:35.388267 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:35.390818 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:35.391516 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:35.391522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:35.391526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:35.391678 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:35.392041 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:35.392046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:35.392049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:35.392390 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:35.392523 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:35.393184 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 20:56:35.393412 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:56:35.394690 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:35.376621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:39691)
0929 20:56:35.377378 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:35.377424 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:35.379145 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:35.379776 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:35.379784 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:35.380318 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:35.380896 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:35.380913 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.381988 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.381992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:35.383467 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:35.383482 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.384270 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.384277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:35.384544 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:35.385303 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:35.385337 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:35.387979 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:35.390960 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:35.390967 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:35.390981 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:35.391391 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:35.391769 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:35.391772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:35.391774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:35.391940 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:35.392493 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:35.392499 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:35.392823 (+ 324us) client_negotiation.cc:769] Sending connection context
0929 20:56:35.393074 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 20:56:35.393464 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":302,"threads_started":1}
I20260929 20:56:35.396323 16182 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37141
I20260929 20:56:35.396646 16182 heartbeater.cc:464] Registering TS with master...
I20260929 20:56:35.397485 16182 heartbeater.cc:511] Master 127.0.97.62:37141 requested a full tablet report, sending...
I20260929 20:56:35.399008 15859 ts_manager.cc:194] Registered new tserver with Master: a05ff9f2ec4e48a6a3510f7de11fb9de (127.0.97.2:33027)
I20260929 20:56:35.399878 15859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39691
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:56:35.432927 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37141
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:56:35.752260 16187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:35.752544 16187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:35.752662 16187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:35.759941 16187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:35.760054 16187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:35.760197 16187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:56:35.769873 16187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:35.771225 16187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:35.772497 16187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:35.785776 16192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:35.785707 16195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:35.785739 16193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:35.786394 16187 server_base.cc:1034] running on GCE node
I20260929 20:56:35.787086 16187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:35.788157 16187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:35.789363 16187 hybrid_clock.cc:648] HybridClock initialized: now 1790715395789306 us; error 49 us; skew 500 ppm
I20260929 20:56:35.789697 16187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:35 dist-test-slave-m9g1 krb5kdc[15814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:35.792783 16187 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:56:35.794281 16187 webserver.cc:460] Webserver started at http://127.0.97.3:41287/ using document root <none> and password file <none>
I20260929 20:56:35.794937 16187 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:35.795035 16187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:35.795287 16187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:56:35.798242 16187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "c8863f58ee6048fa991666f283747135"
format_stamp: "Formatted at 2026-09-29 20:56:35 on dist-test-slave-m9g1"
server_key: "bf8df371bbd5bbbcc0752f822991dae7"
server_key_iv: "0f72bf91c6922dc9015b0715c9588716"
server_key_version: "encryptionkey@0"
I20260929 20:56:35.798998 16187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c8863f58ee6048fa991666f283747135"
format_stamp: "Formatted at 2026-09-29 20:56:35 on dist-test-slave-m9g1"
server_key: "bf8df371bbd5bbbcc0752f822991dae7"
server_key_iv: "0f72bf91c6922dc9015b0715c9588716"
server_key_version: "encryptionkey@0"
I20260929 20:56:35.804065 16187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:56:35.807514 16202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:35.808874 16187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:56:35.809031 16187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "c8863f58ee6048fa991666f283747135"
format_stamp: "Formatted at 2026-09-29 20:56:35 on dist-test-slave-m9g1"
server_key: "bf8df371bbd5bbbcc0752f822991dae7"
server_key_iv: "0f72bf91c6922dc9015b0715c9588716"
server_key_version: "encryptionkey@0"
I20260929 20:56:35.809217 16187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:35.842070 16187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:35.846534 16187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:35.846819 16187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:35.847795 16187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:56:35.849172 16187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:56:35.849257 16187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:35.849380 16187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:56:35.849432 16187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:35.878683 16187 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34559
I20260929 20:56:35.878732 16314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34559 every 8 connection(s)
I20260929 20:56:35.880008 16187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:56:35.889392 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 16187
I20260929 20:56:35.889573 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:56:35.890146 388 external_mini_cluster.cc:1428] Setting key 95a7d95b91ff9196ea5f05a803bbf0cd
Sep 29 20:56:35 dist-test-slave-m9g1 krb5kdc[15814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:56:35.900586 16050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:35.882529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53391 (local address 127.0.97.62:37141)
0929 20:56:35.882694 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:35.882699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:35.883841 (+ 1142us) server_negotiation.cc:407] Connection header received
0929 20:56:35.885647 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:35.885653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:35.885739 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:35.885825 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:35.887393 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.888155 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.890225 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.890811 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.890953 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:35.894843 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:35.894848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:35.894851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:35.894885 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:35.897431 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:35.898413 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:35.898419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:35.898423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:35.898582 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:35.898997 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:35.899000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:35.899002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:35.899277 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:35.899387 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:35.900128 (+ 741us) server_negotiation.cc:299] Negotiation successful
0929 20:56:35.900320 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:56:35.901616 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:35.883013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:53391)
0929 20:56:35.883685 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:35.883723 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:35.885368 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:35.886079 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:35.886090 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:35.886652 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:35.887223 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:35.887242 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.888309 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.888313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:35.890020 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:35.890037 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.891057 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.891066 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:35.891361 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:35.892087 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:35.892112 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:35.894651 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:35.897586 (+ 2935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:35.897597 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:35.897611 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:35.898270 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:35.898675 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:35.898678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:35.898681 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:35.898910 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:35.899370 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:35.899376 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:35.899746 (+ 370us) client_negotiation.cc:769] Sending connection context
0929 20:56:35.900007 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 20:56:35.900478 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":269,"threads_started":1}
I20260929 20:56:35.903255 16315 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37141
I20260929 20:56:35.903577 16315 heartbeater.cc:464] Registering TS with master...
I20260929 20:56:35.904404 16315 heartbeater.cc:511] Master 127.0.97.62:37141 requested a full tablet report, sending...
I20260929 20:56:35.906073 15858 ts_manager.cc:194] Registered new tserver with Master: c8863f58ee6048fa991666f283747135 (127.0.97.3:34559)
I20260929 20:56:35.907024 15858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53391
I20260929 20:56:35.918568 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:56:35.922111 16048 heartbeater.cc:503] Master 127.0.97.62:37141 was elected leader, sending a full tablet report...
I20260929 20:56:35.934839 16050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:35.922904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.0.97.62:37141)
0929 20:56:35.923083 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:35.923089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:35.923613 (+ 524us) server_negotiation.cc:407] Connection header received
0929 20:56:35.923779 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:35.923783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:35.923872 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:35.924012 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:35.925274 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.925978 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.927346 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.927741 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.927841 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:35.930025 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:35.930030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:35.930033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:35.930062 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:35.932553 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:35.933186 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:35.933190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:35.933192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:35.933324 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:35.933622 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:35.933625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:35.933627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:35.933960 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:35.934146 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:35.934438 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 20:56:35.934617 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:56:35.938666 15858 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:56:35.945291 15858 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:56:35.967219 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:35.956903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.0.97.1:45865)
0929 20:56:35.957479 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:35.957488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:35.958640 (+ 1152us) server_negotiation.cc:407] Connection header received
0929 20:56:35.958761 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:35.958769 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:35.959018 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:35.959229 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:35.960116 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.961781 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.964559 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.965989 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.966132 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:35.966382 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:35.966709 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 20:56:35.966875 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":246,"threads_started":1}
I20260929 20:56:35.967239 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:35.956789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45865 (local address 127.0.0.1:43984)
0929 20:56:35.958270 (+ 1481us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:35.958304 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:35.958466 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:35.959503 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:35.959511 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:35.959542 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:35.959919 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:35.959930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.961960 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.961966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:35.964378 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:35.964394 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.966229 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.966236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:35.966429 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:35.966483 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 20:56:35.966646 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 20:56:35.966855 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1224,"thread_start_us":1099,"threads_started":1}
I20260929 20:56:35.967895 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:35.957074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33027 (local address 127.0.0.1:45346)
0929 20:56:35.959021 (+ 1947us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:35.959038 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:35.959159 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:35.960562 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:35.960567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:35.960605 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:35.961012 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:35.961023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.963052 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.963059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:35.965318 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:35.965332 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.967196 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.967202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:35.967372 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:35.967403 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:56:35.967480 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 20:56:35.967595 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1805,"thread_start_us":1677,"threads_started":1}
I20260929 20:56:35.968601 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:35.957639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45346 (local address 127.0.97.2:33027)
0929 20:56:35.958592 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:35.958601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:35.959272 (+ 671us) server_negotiation.cc:407] Connection header received
0929 20:56:35.959377 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:35.959383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:35.959600 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:35.959784 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:35.961210 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.962867 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.965489 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.966759 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.966867 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:35.967056 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:35.968113 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0929 20:56:35.968276 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":811,"thread_start_us":690,"threads_started":1}
I20260929 20:56:35.971503 15983 tablet_service.cc:1467] Processing CreateTablet for tablet c97ba866b1ab4c76bf7190a3bcc48369 (DEFAULT_TABLE table=test-table [id=3956204730234af1addc97698a07628f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:56:35.971539 16117 tablet_service.cc:1467] Processing CreateTablet for tablet c97ba866b1ab4c76bf7190a3bcc48369 (DEFAULT_TABLE table=test-table [id=3956204730234af1addc97698a07628f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:56:35.972877 15983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97ba866b1ab4c76bf7190a3bcc48369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:35.973011 16117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97ba866b1ab4c76bf7190a3bcc48369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:35.974422 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:35.958694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34559 (local address 127.0.0.1:48872)
0929 20:56:35.965318 (+ 6624us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:35.965336 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:35.965453 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:35.966459 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:35.966464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:35.966483 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:35.966856 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:35.966867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.969461 (+ 2594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.969467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:35.971424 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:35.971440 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.973591 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.973598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:35.973769 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:35.973801 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:56:35.973928 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 20:56:35.974066 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6499,"thread_start_us":6369,"threads_started":1}
I20260929 20:56:35.974653 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:35.959225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48872 (local address 127.0.97.3:34559)
0929 20:56:35.965663 (+ 6438us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:35.965673 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:35.965704 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:56:35.965795 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:35.965802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:35.966019 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:35.966214 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:35.967046 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.969312 (+ 2266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.971611 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:35.973082 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:35.973212 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:35.973451 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:35.973987 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 20:56:35.974150 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6249,"thread_start_us":6287,"threads_started":1}
I20260929 20:56:35.978289 16250 tablet_service.cc:1467] Processing CreateTablet for tablet c97ba866b1ab4c76bf7190a3bcc48369 (DEFAULT_TABLE table=test-table [id=3956204730234af1addc97698a07628f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:56:35.979957 16250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97ba866b1ab4c76bf7190a3bcc48369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:35.984287 16333 tablet_bootstrap.cc:492] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb: Bootstrap starting.
I20260929 20:56:35.987736 16334 tablet_bootstrap.cc:492] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de: Bootstrap starting.
I20260929 20:56:35.988775 16333 tablet_bootstrap.cc:654] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:35.990085 16333 log.cc:824] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:35.991171 16334 tablet_bootstrap.cc:654] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:35.992367 16334 log.cc:824] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:35.993417 16333 tablet_bootstrap.cc:492] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb: No bootstrap required, opened a new log
I20260929 20:56:35.993714 16333 ts_tablet_manager.cc:1397] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260929 20:56:35.993839 16337 tablet_bootstrap.cc:492] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135: Bootstrap starting.
I20260929 20:56:35.995651 16334 tablet_bootstrap.cc:492] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de: No bootstrap required, opened a new log
I20260929 20:56:35.995894 16334 ts_tablet_manager.cc:1397] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 20:56:35.997993 16337 tablet_bootstrap.cc:654] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:35.999315 16337 log.cc:824] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:36.002535 16337 tablet_bootstrap.cc:492] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135: No bootstrap required, opened a new log
I20260929 20:56:36.002807 16337 ts_tablet_manager.cc:1397] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 20:56:36.003504 16334 raft_consensus.cc:348] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } }
I20260929 20:56:36.003863 16334 raft_consensus.cc:374] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:36.003955 16334 raft_consensus.cc:729] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05ff9f2ec4e48a6a3510f7de11fb9de, State: Initialized, Role: FOLLOWER
I20260929 20:56:36.003990 16333 raft_consensus.cc:348] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } }
I20260929 20:56:36.004415 16333 raft_consensus.cc:374] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:36.004508 16333 raft_consensus.cc:729] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088f9fb429d34b12b20ab1e6615787cb, State: Initialized, Role: FOLLOWER
I20260929 20:56:36.004561 16334 consensus_queue.cc:260] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } }
I20260929 20:56:36.005054 16333 consensus_queue.cc:260] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } }
I20260929 20:56:36.011543 16182 heartbeater.cc:503] Master 127.0.97.62:37141 was elected leader, sending a full tablet report...
I20260929 20:56:36.011718 16333 ts_tablet_manager.cc:1428] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb: Time spent starting tablet: real 0.018s user 0.009s sys 0.009s
I20260929 20:56:36.012593 16334 ts_tablet_manager.cc:1428] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de: Time spent starting tablet: real 0.017s user 0.009s sys 0.008s
I20260929 20:56:36.014281 16337 raft_consensus.cc:348] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } }
I20260929 20:56:36.014707 16337 raft_consensus.cc:374] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:36.014808 16337 raft_consensus.cc:729] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8863f58ee6048fa991666f283747135, State: Initialized, Role: FOLLOWER
I20260929 20:56:36.015511 16337 consensus_queue.cc:260] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } }
I20260929 20:56:36.016885 16315 heartbeater.cc:503] Master 127.0.97.62:37141 was elected leader, sending a full tablet report...
I20260929 20:56:36.017508 16337 ts_tablet_manager.cc:1428] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
W20260929 20:56:36.125524 16183 tablet.cc:2369] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:56:36.131626 16316 tablet.cc:2369] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:56:36.143054 16049 tablet.cc:2369] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:56:36.181026 16340 raft_consensus.cc:482] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:56:36.181327 16340 raft_consensus.cc:504] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } }
I20260929 20:56:36.183219 16340 leader_election.cc:288] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a05ff9f2ec4e48a6a3510f7de11fb9de (127.0.97.2:33027), c8863f58ee6048fa991666f283747135 (127.0.97.3:34559)
I20260929 20:56:36.192070 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:36.183699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43247 (local address 127.0.97.3:34559)
0929 20:56:36.183899 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:36.183919 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:36.184673 (+ 754us) server_negotiation.cc:407] Connection header received
0929 20:56:36.184828 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:36.184833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:36.184906 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:36.184990 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:36.185658 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:36.186749 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:36.189614 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:36.190853 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:36.190986 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:36.191095 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:36.191674 (+ 579us) server_negotiation.cc:299] Negotiation successful
0929 20:56:36.191813 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:56:36.192225 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:36.183692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34559 (local address 127.0.97.1:43247)
0929 20:56:36.184548 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:36.184566 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:36.184695 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:36.185081 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:36.185089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:36.185133 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:36.185519 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:36.185528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:36.186892 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:36.186896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:36.189465 (+ 2569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:36.189483 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:36.191143 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:36.191159 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:36.191493 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:36.191518 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 20:56:36.191613 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 20:56:36.191813 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":234,"spinlock_wait_cycles":7808,"thread_start_us":415,"threads_started":1}
I20260929 20:56:36.192346 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:36.183683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49127 (local address 127.0.97.2:33027)
0929 20:56:36.183888 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:36.183895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:36.184562 (+ 667us) server_negotiation.cc:407] Connection header received
0929 20:56:36.184745 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:36.184751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:36.184827 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:36.184901 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:36.185632 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:36.186746 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:36.189610 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:36.190853 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:36.190986 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:36.191095 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:36.191954 (+ 859us) server_negotiation.cc:299] Negotiation successful
0929 20:56:36.192119 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:56:36.192831 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:36.183588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33027 (local address 127.0.97.1:49127)
0929 20:56:36.184449 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:36.184468 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:36.184607 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:36.184934 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:36.184939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:36.184961 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:36.185468 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:36.185490 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:36.186889 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:36.186894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:36.189465 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:36.189483 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:36.191152 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:36.191158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:36.191429 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:36.191486 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 20:56:36.191604 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 20:56:36.191813 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":184,"spinlock_wait_cycles":31488,"thread_start_us":485,"threads_started":1}
I20260929 20:56:36.193331 16270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97ba866b1ab4c76bf7190a3bcc48369" candidate_uuid: "088f9fb429d34b12b20ab1e6615787cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8863f58ee6048fa991666f283747135" is_pre_election: true
I20260929 20:56:36.193879 16270 raft_consensus.cc:2454] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088f9fb429d34b12b20ab1e6615787cb in term 0.
I20260929 20:56:36.193928 16137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97ba866b1ab4c76bf7190a3bcc48369" candidate_uuid: "088f9fb429d34b12b20ab1e6615787cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" is_pre_election: true
I20260929 20:56:36.194505 16137 raft_consensus.cc:2454] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088f9fb429d34b12b20ab1e6615787cb in term 0.
I20260929 20:56:36.194612 15939 leader_election.cc:302] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088f9fb429d34b12b20ab1e6615787cb, c8863f58ee6048fa991666f283747135; no voters:
I20260929 20:56:36.195127 16340 raft_consensus.cc:2781] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:56:36.195294 16340 raft_consensus.cc:482] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:56:36.195400 16340 raft_consensus.cc:3037] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:36.197636 16340 raft_consensus.cc:504] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } }
I20260929 20:56:36.198290 16340 leader_election.cc:288] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [CANDIDATE]: Term 1 election: Requested vote from peers a05ff9f2ec4e48a6a3510f7de11fb9de (127.0.97.2:33027), c8863f58ee6048fa991666f283747135 (127.0.97.3:34559)
I20260929 20:56:36.198652 16137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97ba866b1ab4c76bf7190a3bcc48369" candidate_uuid: "088f9fb429d34b12b20ab1e6615787cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de"
I20260929 20:56:36.198793 16270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97ba866b1ab4c76bf7190a3bcc48369" candidate_uuid: "088f9fb429d34b12b20ab1e6615787cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8863f58ee6048fa991666f283747135"
I20260929 20:56:36.198839 16137 raft_consensus.cc:3037] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:36.198988 16270 raft_consensus.cc:3037] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:36.201107 16137 raft_consensus.cc:2454] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088f9fb429d34b12b20ab1e6615787cb in term 1.
I20260929 20:56:36.201164 16270 raft_consensus.cc:2454] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088f9fb429d34b12b20ab1e6615787cb in term 1.
I20260929 20:56:36.201499 15938 leader_election.cc:302] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088f9fb429d34b12b20ab1e6615787cb, a05ff9f2ec4e48a6a3510f7de11fb9de; no voters:
I20260929 20:56:36.201776 16340 raft_consensus.cc:2781] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:56:36.202214 16340 raft_consensus.cc:686] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [term 1 LEADER]: Becoming Leader. State: Replica: 088f9fb429d34b12b20ab1e6615787cb, State: Running, Role: LEADER
I20260929 20:56:36.202744 16340 consensus_queue.cc:237] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } }
I20260929 20:56:36.207033 15858 catalog_manager.cc:5495] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 088f9fb429d34b12b20ab1e6615787cb (127.0.97.1). New cstate: current_term: 1 leader_uuid: "088f9fb429d34b12b20ab1e6615787cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } health_report { overall_health: UNKNOWN } } }
I20260929 20:56:36.263127 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:36.257324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43990 (local address 127.0.97.1:45865)
0929 20:56:36.257492 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:36.257496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:36.257513 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:56:36.257607 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:36.257610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:36.257691 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:36.257811 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:56:36.258212 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:36.258908 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:36.260925 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:36.261355 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:36.261448 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:36.261978 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:56:36.262575 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:56:36.262683 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:36.262801 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 20:56:36.262870 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:56:36.271270 16137 raft_consensus.cc:1261] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de [term 1 FOLLOWER]: Refusing update from remote peer 088f9fb429d34b12b20ab1e6615787cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:56:36.271270 16270 raft_consensus.cc:1261] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135 [term 1 FOLLOWER]: Refusing update from remote peer 088f9fb429d34b12b20ab1e6615787cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:56:36.272145 16344 consensus_queue.cc:1035] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:56:36.272495 16340 consensus_queue.cc:1035] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:56:36.297030 15859 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:56:36.297307 15859 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:56:36.300915 15859 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245: Sending DeleteTablet for 3 replicas of tablet c97ba866b1ab4c76bf7190a3bcc48369
I20260929 20:56:36.302399 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 15920
I20260929 20:56:36.302300 16117 tablet_service.cc:1514] Processing DeleteTablet for tablet c97ba866b1ab4c76bf7190a3bcc48369 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:56:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45346
I20260929 20:56:36.302438 16250 tablet_service.cc:1514] Processing DeleteTablet for tablet c97ba866b1ab4c76bf7190a3bcc48369 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:56:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48872
I20260929 20:56:36.303067 15983 tablet_service.cc:1514] Processing DeleteTablet for tablet c97ba866b1ab4c76bf7190a3bcc48369 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:56:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43984
Sep 29 20:56:36 dist-test-slave-m9g1 krb5kdc[15814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:56:36.315698 16358 tablet_replica.cc:330] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de: stopping tablet replica
I20260929 20:56:36.316219 16358 raft_consensus.cc:2229] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:56:36.316810 16358 raft_consensus.cc:2258] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:56:36.317566 16357 tablet_replica.cc:330] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135: stopping tablet replica
I20260929 20:56:36.318559 16357 raft_consensus.cc:2229] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:56:36.319156 16357 raft_consensus.cc:2258] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:56:36.322345 16360 tablet_replica.cc:330] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb: stopping tablet replica
I20260929 20:56:36.322911 16360 raft_consensus.cc:2229] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:56:36.323001 16357 ts_tablet_manager.cc:1905] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:56:36.323773 16360 raft_consensus.cc:2258] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:56:36.324870 16358 ts_tablet_manager.cc:1905] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:56:36.325659 16360 ts_tablet_manager.cc:1905] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:56:36.326020 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:36.303734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44002 (local address 127.0.97.1:45865)
0929 20:56:36.303916 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:36.303922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:36.303944 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:56:36.304009 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:36.304014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:36.304120 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:36.304241 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:36.306097 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:36.307726 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:36.309829 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:36.310557 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:36.310663 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:36.317154 (+ 6491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:36.317175 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:36.317196 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:36.317256 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:36.321432 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:36.322546 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:36.322556 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:36.322560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:36.322777 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:36.323511 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:36.323516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:36.323519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:36.324186 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:36.324510 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:36.325245 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 20:56:36.325515 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:56:36.327769 16043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:36.333297 16357 ts_tablet_manager.cc:1918] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:56:36.333479 16357 log.cc:1196] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/c97ba866b1ab4c76bf7190a3bcc48369
I20260929 20:56:36.333907 16357 ts_tablet_manager.cc:1939] T c97ba866b1ab4c76bf7190a3bcc48369 P c8863f58ee6048fa991666f283747135: Deleting consensus metadata
I20260929 20:56:36.335685 15846 catalog_manager.cc:4841] TS c8863f58ee6048fa991666f283747135 (127.0.97.3:34559): tablet c97ba866b1ab4c76bf7190a3bcc48369 (table test-table [id=3956204730234af1addc97698a07628f]) successfully deleted
I20260929 20:56:36.336180 16358 ts_tablet_manager.cc:1918] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:56:36.336366 16358 log.cc:1196] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/c97ba866b1ab4c76bf7190a3bcc48369
I20260929 20:56:36.336881 16358 ts_tablet_manager.cc:1939] T c97ba866b1ab4c76bf7190a3bcc48369 P a05ff9f2ec4e48a6a3510f7de11fb9de: Deleting consensus metadata
I20260929 20:56:36.339039 15845 catalog_manager.cc:4841] TS a05ff9f2ec4e48a6a3510f7de11fb9de (127.0.97.2:33027): tablet c97ba866b1ab4c76bf7190a3bcc48369 (table test-table [id=3956204730234af1addc97698a07628f]) successfully deleted
I20260929 20:56:36.428493 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 15920
W20260929 20:56:36.443756 15846 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45865 (error 108)
I20260929 20:56:36.444006 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 16054
I20260929 20:56:36.445405 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:36.444952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45865 (local address 127.0.0.1:44008)
0929 20:56:36.445132 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:36.445234 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 20:56:36.445690 15846 catalog_manager.cc:4584] TS 088f9fb429d34b12b20ab1e6615787cb (127.0.97.1:45865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c97ba866b1ab4c76bf7190a3bcc48369: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45865: connect: Connection refused (error 111)
Sep 29 20:56:36 dist-test-slave-m9g1 krb5kdc[15814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:56:36.458774 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:36.444709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45358 (local address 127.0.97.2:33027)
0929 20:56:36.444879 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:36.444883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:36.445270 (+ 387us) server_negotiation.cc:407] Connection header received
0929 20:56:36.445411 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:36.445417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:36.445503 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:36.445626 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:36.446918 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:36.447647 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:36.448892 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:36.449274 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:36.449381 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:36.453524 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:36.453538 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:36.453553 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:36.453598 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:36.456159 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:36.456781 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:36.456788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:36.456791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:36.456909 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:36.457249 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:36.457252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:36.457255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:36.457631 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:36.457897 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:36.458239 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 20:56:36.458450 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:56:36.459430 16177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:36.499287 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:36.498798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45865 (local address 127.0.0.1:44018)
0929 20:56:36.499026 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:36.499138 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 20:56:36.500346 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:36.499969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45865 (local address 127.0.0.1:44022)
0929 20:56:36.500126 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:36.500200 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 20:56:36.549748 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 16054
I20260929 20:56:36.561009 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:36.560410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45865 (local address 127.0.0.1:44024)
0929 20:56:36.560671 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:36.560789 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 20:56:36.562151 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:36.561724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45865 (local address 127.0.0.1:44032)
0929 20:56:36.561872 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:36.561994 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 20:56:36.564128 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 16187
Sep 29 20:56:36 dist-test-slave-m9g1 krb5kdc[15814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:56:36.580731 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:36.564898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48882 (local address 127.0.97.3:34559)
0929 20:56:36.565077 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:36.565082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:36.565103 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:56:36.565229 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:36.565233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:36.565308 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:36.565395 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:36.566709 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:36.567418 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:36.568830 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:36.569231 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:36.569352 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:36.574739 (+ 5387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:36.574761 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:36.574782 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:36.574827 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:36.578132 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:36.578862 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:36.578870 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:36.578872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:36.579008 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:36.579370 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:36.579373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:36.579375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:36.579716 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:36.579922 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:36.580294 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 20:56:36.580471 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:56:36.581385 16310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:36.643034 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:36.642469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45865 (local address 127.0.0.1:44034)
0929 20:56:36.642732 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:36.642850 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260929 20:56:36.644100 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:36.643715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45865 (local address 127.0.0.1:44038)
0929 20:56:36.643912 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:36.643989 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 20:56:36.671132 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 16187
I20260929 20:56:36.686775 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 15827
I20260929 20:56:36.687880 15889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:36.796941 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 15827
I20260929 20:56:56.814989 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37141
--webserver_interface=127.0.97.62
--webserver_port=38113
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:56:57.138499 16366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:57.138798 16366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:57.138923 16366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:57.146152 16366 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 20:56:57.146306 16366 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 20:56:57.146374 16366 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 20:56:57.146448 16366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:56:57.146524 16366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:57.146618 16366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:57.146678 16366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:56:57.146751 16366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:56:57.156821 16366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:57.158176 16366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:57.159582 16366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:57.180840 16372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:57.180791 16375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:57.180878 16373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:57.183677 16366 server_base.cc:1034] running on GCE node
I20260929 20:56:57.184566 16366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:57.186112 16366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:57.187387 16366 hybrid_clock.cc:648] HybridClock initialized: now 1790715417187330 us; error 57 us; skew 500 ppm
I20260929 20:56:57.187755 16366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:57 dist-test-slave-m9g1 krb5kdc[15814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:57.196273 16366 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:56:57.198014 16366 webserver.cc:460] Webserver started at http://127.0.97.62:38113/ using document root <none> and password file <none>
I20260929 20:56:57.198868 16366 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:57.198976 16366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:57.205016 16366 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:56:57.208163 16382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:57.209627 16366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:56:57.209784 16366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "f4707c4c1d364f9aa579a7f8c75be245"
format_stamp: "Formatted at 2026-09-29 20:56:34 on dist-test-slave-m9g1"
server_key: "f978a6b74d0b582be12154a001318628"
server_key_iv: "68e7bb4f2566df506303fc4036c367b2"
server_key_version: "encryptionkey@0"
I20260929 20:56:57.210927 16366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:57.238032 16366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:57.242720 16366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:57.242976 16366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:57.260667 16366 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37141
I20260929 20:56:57.260699 16433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37141 every 8 connection(s)
I20260929 20:56:57.262382 16366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:56:57.267566 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 16366
I20260929 20:56:57.268556 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45865
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37141
--webserver_port=43945
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:56:57.270417 16434 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:56:57.274924 16434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245: Bootstrap starting.
I20260929 20:56:57.295009 16434 log.cc:824] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:57.307211 16434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:56:57.307757 16434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245: Bootstrap complete.
I20260929 20:56:57.317204 16434 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4707c4c1d364f9aa579a7f8c75be245" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } }
I20260929 20:56:57.318089 16434 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4707c4c1d364f9aa579a7f8c75be245, State: Initialized, Role: FOLLOWER
I20260929 20:56:57.318749 16434 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4707c4c1d364f9aa579a7f8c75be245" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } }
I20260929 20:56:57.318972 16434 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:56:57.319097 16434 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:56:57.319268 16434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:56:57.323450 16434 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4707c4c1d364f9aa579a7f8c75be245" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } }
I20260929 20:56:57.323982 16434 leader_election.cc:302] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4707c4c1d364f9aa579a7f8c75be245; no voters:
I20260929 20:56:57.324572 16434 leader_election.cc:288] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:56:57.324709 16438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:56:57.325780 16438 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 2 LEADER]: Becoming Leader. State: Replica: f4707c4c1d364f9aa579a7f8c75be245, State: Running, Role: LEADER
I20260929 20:56:57.326439 16438 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4707c4c1d364f9aa579a7f8c75be245" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } }
I20260929 20:56:57.326771 16434 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:56:57.329002 16440 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4707c4c1d364f9aa579a7f8c75be245. Latest consensus state: current_term: 2 leader_uuid: "f4707c4c1d364f9aa579a7f8c75be245" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4707c4c1d364f9aa579a7f8c75be245" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } }
I20260929 20:56:57.329036 16439 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f4707c4c1d364f9aa579a7f8c75be245" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4707c4c1d364f9aa579a7f8c75be245" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } }
I20260929 20:56:57.329381 16440 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [sys.catalog]: This master's current role is: LEADER
I20260929 20:56:57.329432 16439 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [sys.catalog]: This master's current role is: LEADER
I20260929 20:56:57.347172 16444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:56:57.355067 16444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:56:57.356614 16444 catalog_manager.cc:1259] Loaded cluster ID: ae20801ac44f41adaaef378b12656ebe
I20260929 20:56:57.356743 16444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:56:57.360725 16444 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:56:57.362253 16444 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245: Loaded TSK: 0
I20260929 20:56:57.398080 16444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245: Generated new TSK 1
I20260929 20:56:57.398545 16444 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 20:56:57.622231 16436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:57.622516 16436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:57.622634 16436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:57.630028 16436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:57.630138 16436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:57.630311 16436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:56:57.640266 16436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43945
--tserver_master_addrs=127.0.97.62:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:57.641670 16436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:57.643117 16436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:57.658555 16462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:57.658772 16464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:57.659082 16461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:57.659057 16436 server_base.cc:1034] running on GCE node
I20260929 20:56:57.659757 16436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:57.660743 16436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:57.661947 16436 hybrid_clock.cc:648] HybridClock initialized: now 1790715417661882 us; error 53 us; skew 500 ppm
I20260929 20:56:57.662307 16436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:57 dist-test-slave-m9g1 krb5kdc[15814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:57.665634 16436 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:56:57.667229 16436 webserver.cc:460] Webserver started at http://127.0.97.1:43945/ using document root <none> and password file <none>
I20260929 20:56:57.667912 16436 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:57.668040 16436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:57.674139 16436 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:56:57.677282 16471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:57.678812 16436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:56:57.678964 16436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "088f9fb429d34b12b20ab1e6615787cb"
format_stamp: "Formatted at 2026-09-29 20:56:34 on dist-test-slave-m9g1"
server_key: "fc00edefbfa8aaa12f6e0b9eff507e7c"
server_key_iv: "4f41073b8b11f95d76740283b051a87c"
server_key_version: "encryptionkey@0"
I20260929 20:56:57.679956 16436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:57.695531 16436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:57.699913 16436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:57.700181 16436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:57.701148 16436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:56:57.703073 16478 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 20:56:57.708693 16436 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 20:56:57.708812 16436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 20:56:57.708932 16436 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 20:56:57.712452 16436 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 20:56:57.712545 16436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260929 20:56:57.712889 16478 tablet_bootstrap.cc:492] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb: Bootstrap starting.
I20260929 20:56:57.721515 16478 log.cc:824] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:57.735176 16478 tablet_bootstrap.cc:492] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:56:57.735805 16478 tablet_bootstrap.cc:492] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb: Bootstrap complete.
I20260929 20:56:57.736233 16478 ts_tablet_manager.cc:1397] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb: Time spent bootstrapping tablet: real 0.024s user 0.021s sys 0.000s
I20260929 20:56:57.744208 16478 raft_consensus.cc:348] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } }
I20260929 20:56:57.745184 16478 raft_consensus.cc:729] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088f9fb429d34b12b20ab1e6615787cb, State: Initialized, Role: FOLLOWER
I20260929 20:56:57.745903 16478 consensus_queue.cc:260] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } }
I20260929 20:56:57.748126 16478 ts_tablet_manager.cc:1428] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260929 20:56:57.756886 16436 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45865
I20260929 20:56:57.756944 16586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45865 every 8 connection(s)
I20260929 20:56:57.758061 16436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:56:57.764652 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 16436
I20260929 20:56:57.765393 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33027
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37141
--webserver_port=33685
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:56:57 dist-test-slave-m9g1 krb5kdc[15814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:56:57.781586 16590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:57.760743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:39185)
0929 20:56:57.761538 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:57.761580 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:57.762085 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:57.763558 (+ 1473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:57.763566 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:57.764275 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:57.765169 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:57.765199 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:57.766942 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:57.766947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:57.768659 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:57.768673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:57.769646 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:57.769653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:57.769945 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:57.770639 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:57.770674 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:57.773421 (+ 2747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:57.777614 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:57.777622 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:57.777635 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:57.778107 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:57.778596 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:57.778600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:57.778602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:57.778765 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:57.779643 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:57.779649 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:57.779979 (+ 330us) client_negotiation.cc:769] Sending connection context
0929 20:56:57.780216 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 20:56:57.780612 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":328,"threads_started":1}
I20260929 20:56:57.781960 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:57.760799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39185 (local address 127.0.97.62:37141)
0929 20:56:57.761325 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:57.761336 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:57.761709 (+ 373us) server_negotiation.cc:407] Connection header received
0929 20:56:57.762620 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:57.762635 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:57.763011 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:57.763349 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:57.765400 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:57.766772 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:57.768798 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:57.769460 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:57.769595 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:57.773796 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:57.773807 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:57.773829 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:57.773878 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:57.777454 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:57.778267 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:57.778276 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:57.778286 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:57.778491 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:57.778863 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:57.778867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:57.778870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:57.779517 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:57.779690 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:57.780456 (+ 766us) server_negotiation.cc:299] Negotiation successful
0929 20:56:57.780906 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":229,"threads_started":1}
I20260929 20:56:57.784767 16587 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37141
I20260929 20:56:57.785099 16587 heartbeater.cc:464] Registering TS with master...
I20260929 20:56:57.785779 16587 heartbeater.cc:511] Master 127.0.97.62:37141 requested a full tablet report, sending...
I20260929 20:56:57.788867 16399 ts_manager.cc:194] Registered new tserver with Master: 088f9fb429d34b12b20ab1e6615787cb (127.0.97.1:45865)
I20260929 20:56:57.793195 16399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39185
I20260929 20:56:57.802148 16593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:57.792358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45865 (local address 127.0.0.1:52266)
0929 20:56:57.795532 (+ 3174us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:57.795577 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:57.795785 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:57.796471 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:57.796479 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:57.796509 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:57.796905 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:57.796919 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:57.798371 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:57.798376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:57.799712 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:57.799725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:57.801219 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:57.801226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:57.801420 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:57.801472 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 20:56:57.801600 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 20:56:57.801795 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2946,"thread_start_us":2706,"threads_started":1}
I20260929 20:56:57.802322 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:57.792388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52266 (local address 127.0.97.1:45865)
0929 20:56:57.794893 (+ 2505us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:57.794903 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:57.795664 (+ 761us) server_negotiation.cc:407] Connection header received
0929 20:56:57.795982 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:57.795988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:57.796288 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:57.796444 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:57.797144 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:57.798262 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:57.799871 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:57.801006 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:57.801141 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:57.801746 (+ 605us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:57.801818 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 20:56:57.801974 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2310,"thread_start_us":2133,"threads_started":1}
I20260929 20:56:57.805680 16521 tablet_service.cc:1514] Processing DeleteTablet for tablet c97ba866b1ab4c76bf7190a3bcc48369 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:56:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52266
I20260929 20:56:57.808296 16594 tablet_replica.cc:330] stopping tablet replica
I20260929 20:56:57.808755 16594 raft_consensus.cc:2229] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:56:57.809207 16594 raft_consensus.cc:2258] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:56:57.810421 16594 ts_tablet_manager.cc:1905] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:56:57.817633 16594 ts_tablet_manager.cc:1918] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:56:57.817767 16594 log.cc:1196] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/c97ba866b1ab4c76bf7190a3bcc48369
I20260929 20:56:57.818151 16594 ts_tablet_manager.cc:1939] T c97ba866b1ab4c76bf7190a3bcc48369 P 088f9fb429d34b12b20ab1e6615787cb: Deleting consensus metadata
I20260929 20:56:57.820261 16386 catalog_manager.cc:4841] TS 088f9fb429d34b12b20ab1e6615787cb (127.0.97.1:45865): tablet c97ba866b1ab4c76bf7190a3bcc48369 (table test-table [id=3956204730234af1addc97698a07628f]) successfully deleted
W20260929 20:56:58.099109 16591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:58.099426 16591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:58.099529 16591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:58.106848 16591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:58.106971 16591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:58.107091 16591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:56:58.117870 16591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33685
--tserver_master_addrs=127.0.97.62:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:58.119225 16591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:58.120556 16591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:58.133517 16599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:58.133543 16600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:58.133579 16602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:58.134310 16591 server_base.cc:1034] running on GCE node
I20260929 20:56:58.134912 16591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:58.135854 16591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:58.137050 16591 hybrid_clock.cc:648] HybridClock initialized: now 1790715418137006 us; error 44 us; skew 500 ppm
I20260929 20:56:58.137362 16591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:58 dist-test-slave-m9g1 krb5kdc[15814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:58.140503 16591 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:56:58.142127 16591 webserver.cc:460] Webserver started at http://127.0.97.2:33685/ using document root <none> and password file <none>
I20260929 20:56:58.142886 16591 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:58.143005 16591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:58.148622 16591 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:56:58.151445 16609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:58.152907 16591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:56:58.153074 16591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de"
format_stamp: "Formatted at 2026-09-29 20:56:35 on dist-test-slave-m9g1"
server_key: "4d7c59596e5058aaa4586c868e60b4dc"
server_key_iv: "83e0f028a35d9ed4125921ea6779368a"
server_key_version: "encryptionkey@0"
I20260929 20:56:58.154091 16591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:58.182798 16591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:58.187314 16591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:58.187579 16591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:58.188547 16591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:56:58.189888 16591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:56:58.189982 16591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:58.190083 16591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:56:58.190151 16591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:58.220191 16591 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33027
I20260929 20:56:58.220273 16721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33027 every 8 connection(s)
I20260929 20:56:58.221689 16591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:56:58.223338 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 16591
I20260929 20:56:58.223917 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34559
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37141
--webserver_port=41287
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:56:58 dist-test-slave-m9g1 krb5kdc[15814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:56:58.242642 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:58.224375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37875 (local address 127.0.97.62:37141)
0929 20:56:58.224587 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:58.224593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:58.225745 (+ 1152us) server_negotiation.cc:407] Connection header received
0929 20:56:58.228269 (+ 2524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:58.228275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:58.228363 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:58.228455 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:58.230575 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.231292 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.233034 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.233458 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.233573 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:58.236990 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:58.236996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:58.236999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:58.237033 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:58.239495 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:58.240220 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:58.240224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:58.240226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:58.240347 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:58.240795 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:58.240799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:58.240801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:58.241134 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:58.241266 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:58.242031 (+ 765us) server_negotiation.cc:299] Negotiation successful
0929 20:56:58.242306 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:56:58.243693 16724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:58.224746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:37875)
0929 20:56:58.225541 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:58.225601 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:58.227815 (+ 2214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:58.228763 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:58.228776 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:58.229539 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:58.230379 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:58.230405 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.231455 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.231460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:58.232873 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:58.232887 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.233681 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.233688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:58.233933 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:58.234588 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:58.234611 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:58.236745 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:58.239631 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:58.239640 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:58.239659 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:58.240091 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:58.240449 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:58.240454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:58.240458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:58.240685 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:58.241258 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:58.241267 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:58.241619 (+ 352us) client_negotiation.cc:769] Sending connection context
0929 20:56:58.241898 (+ 279us) client_negotiation.cc:240] Negotiation successful
0929 20:56:58.242517 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":287,"threads_started":1}
I20260929 20:56:58.245263 16722 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37141
I20260929 20:56:58.245601 16722 heartbeater.cc:464] Registering TS with master...
I20260929 20:56:58.246356 16722 heartbeater.cc:511] Master 127.0.97.62:37141 requested a full tablet report, sending...
I20260929 20:56:58.247831 16399 ts_manager.cc:194] Registered new tserver with Master: a05ff9f2ec4e48a6a3510f7de11fb9de (127.0.97.2:33027)
I20260929 20:56:58.248935 16399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37875
W20260929 20:56:58.548882 16725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:56:58.549160 16725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:56:58.549252 16725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:56:58.556507 16725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:56:58.556624 16725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:56:58.556737 16725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:56:58.566601 16725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41287
--tserver_master_addrs=127.0.97.62:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:56:58.567834 16725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:56:58.569100 16725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:56:58.587873 16731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:56:58.587916 16730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:58.589437 16725 server_base.cc:1034] running on GCE node
W20260929 20:56:58.589533 16733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:56:58.590369 16725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:56:58.591579 16725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:56:58.592818 16725 hybrid_clock.cc:648] HybridClock initialized: now 1790715418592723 us; error 62 us; skew 500 ppm
I20260929 20:56:58.593228 16725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:56:58 dist-test-slave-m9g1 krb5kdc[15814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:56:58.596976 16725 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:56:58.598928 16725 webserver.cc:460] Webserver started at http://127.0.97.3:41287/ using document root <none> and password file <none>
I20260929 20:56:58.599911 16725 fs_manager.cc:359] Metadata directory not provided
I20260929 20:56:58.600080 16725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:56:58.606181 16725 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:56:58.608973 16740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:58.610371 16725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:56:58.610524 16725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "c8863f58ee6048fa991666f283747135"
format_stamp: "Formatted at 2026-09-29 20:56:35 on dist-test-slave-m9g1"
server_key: "bf8df371bbd5bbbcc0752f822991dae7"
server_key_iv: "0f72bf91c6922dc9015b0715c9588716"
server_key_version: "encryptionkey@0"
I20260929 20:56:58.611500 16725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:56:58.647027 16725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:56:58.651528 16725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:56:58.651769 16725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:56:58.652805 16725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:56:58.654172 16725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:56:58.654300 16725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:58.654392 16725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:56:58.654446 16725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:56:58.683027 16725 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34559
I20260929 20:56:58.683100 16852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34559 every 8 connection(s)
I20260929 20:56:58.684549 16725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:56:58.688898 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 16725
Sep 29 20:56:58 dist-test-slave-m9g1 krb5kdc[15814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:56:58.704656 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:58.689853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59118 (local address 127.0.97.62:37141)
0929 20:56:58.690271 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:58.690330 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:58.690623 (+ 293us) server_negotiation.cc:407] Connection header received
0929 20:56:58.690803 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:58.690807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:58.690871 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:58.690962 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:58.692566 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.693342 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.694688 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.695097 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.695207 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:58.698619 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:58.698627 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:58.698632 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:58.698674 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:58.701966 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:58.702782 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:58.702788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:58.702791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:58.702933 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:58.703305 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:58.703308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:58.703310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:58.703583 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:58.703763 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:58.704124 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 20:56:58.704289 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":286,"thread_start_us":208,"threads_started":1}
I20260929 20:56:58.706432 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:58.687186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33037 (local address 127.0.97.62:37141)
0929 20:56:58.687400 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:58.687406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:58.688701 (+ 1295us) server_negotiation.cc:407] Connection header received
0929 20:56:58.690957 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:58.690963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:58.691041 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:58.691126 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:58.692715 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.693616 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.695299 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.695737 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.695854 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:58.700144 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:58.700150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:58.700155 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:58.700208 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:58.703117 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:58.703871 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:58.703875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:58.703877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:58.703974 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:58.704458 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:58.704463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:58.704466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:58.704822 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:58.704958 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:58.705837 (+ 879us) server_negotiation.cc:299] Negotiation successful
0929 20:56:58.706069 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:56:58.707541 16855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:58.687720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:33037)
0929 20:56:58.688498 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:58.688548 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:58.690671 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:58.691417 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:58.691426 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:58.691937 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:56:58.692540 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:58.692557 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.693806 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.693810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:58.695153 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:58.695165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.695962 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.695968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:58.696281 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:58.696974 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:56:58.697005 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:56:58.699915 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:56:58.703246 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:58.703257 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:58.703276 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:58.703727 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:58.704075 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:56:58.704080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:56:58.704083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:56:58.704330 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:56:58.704959 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:56:58.704968 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:56:58.705418 (+ 450us) client_negotiation.cc:769] Sending connection context
0929 20:56:58.705721 (+ 303us) client_negotiation.cc:240] Negotiation successful
0929 20:56:58.706326 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":290,"threads_started":1}
I20260929 20:56:58.708770 16853 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37141
I20260929 20:56:58.709054 16853 heartbeater.cc:464] Registering TS with master...
I20260929 20:56:58.709751 16853 heartbeater.cc:511] Master 127.0.97.62:37141 requested a full tablet report, sending...
I20260929 20:56:58.711153 16399 ts_manager.cc:194] Registered new tserver with Master: c8863f58ee6048fa991666f283747135 (127.0.97.3:34559)
I20260929 20:56:58.711944 16399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33037
I20260929 20:56:58.719619 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:56:58.725971 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:58.720601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59122 (local address 127.0.97.62:37141)
0929 20:56:58.720834 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:58.720839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:58.721172 (+ 333us) server_negotiation.cc:407] Connection header received
0929 20:56:58.721347 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:58.721350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:58.721498 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:58.721620 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:56:58.722002 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.722777 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.724301 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.724673 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.724766 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:58.725239 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:56:58.725447 (+ 208us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 20:56:58.725733 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59122: token expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 20:56:58.726114 16589 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59122: token expired
W20260929 20:56:58.726732 16318 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:37141)
I20260929 20:56:58.739375 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:58.727193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59138 (local address 127.0.97.62:37141)
0929 20:56:58.727343 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:58.727348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:58.727403 (+ 55us) server_negotiation.cc:407] Connection header received
0929 20:56:58.727546 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:58.727551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:58.727614 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:58.727691 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:58.729233 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.729943 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.731325 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.731767 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.731876 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:58.734405 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:58.734409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:58.734413 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:58.734444 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:58.736766 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:58.737421 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:58.737427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:58.737431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:58.737557 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:58.737900 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:58.737905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:58.737909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:58.738275 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:58.738531 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:58.738824 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 20:56:58.739047 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 20:56:58.742729 16399 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:56:58.749826 16399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:56:58.775254 16521 tablet_service.cc:1467] Processing CreateTablet for tablet 4dab07b90aad478192c8c496ccaaabcb (DEFAULT_TABLE table=test-table [id=d4445629e2294e488b4dd8e609e0890b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:56:58.776297 16521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dab07b90aad478192c8c496ccaaabcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:58.784049 16865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:58.760557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37144 (local address 127.0.97.3:34559)
0929 20:56:58.774935 (+ 14378us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:58.774945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:58.774987 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:56:58.775098 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:58.775106 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:58.775403 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:58.775622 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:58.776528 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.778149 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.780588 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.782487 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.783215 (+ 728us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:58.783467 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:58.783550 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 20:56:58.783708 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14182,"thread_start_us":20492,"threads_started":1}
I20260929 20:56:58.784859 16863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:58.760231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34559 (local address 127.0.0.1:37144)
0929 20:56:58.774120 (+ 13889us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:58.774147 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:58.774344 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:58.775952 (+ 1608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:58.775957 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:58.775986 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:58.776299 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:58.776313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.778352 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.778358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:58.780392 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:58.780409 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.782791 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.782798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:58.783003 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:58.783043 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 20:56:58.783127 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 20:56:58.783263 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13710,"thread_start_us":13534,"threads_started":1}
I20260929 20:56:58.785838 16867 tablet_bootstrap.cc:492] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb: Bootstrap starting.
I20260929 20:56:58.787624 16867 tablet_bootstrap.cc:654] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:58.789083 16788 tablet_service.cc:1467] Processing CreateTablet for tablet 4dab07b90aad478192c8c496ccaaabcb (DEFAULT_TABLE table=test-table [id=d4445629e2294e488b4dd8e609e0890b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:56:58.789836 16867 tablet_bootstrap.cc:492] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb: No bootstrap required, opened a new log
I20260929 20:56:58.790040 16867 ts_tablet_manager.cc:1397] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 20:56:58.790527 16788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dab07b90aad478192c8c496ccaaabcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:58.790863 16867 raft_consensus.cc:348] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } }
I20260929 20:56:58.791088 16866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:58.761288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32866 (local address 127.0.97.2:33027)
0929 20:56:58.782696 (+ 21408us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:58.782706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:58.783596 (+ 890us) server_negotiation.cc:407] Connection header received
0929 20:56:58.783718 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:58.783727 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:58.784025 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:58.784217 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:58.785182 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.786399 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.788684 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.789938 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.790078 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:58.790304 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:58.790597 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 20:56:58.790766 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21233,"thread_start_us":21097,"threads_started":1}
I20260929 20:56:58.791134 16867 raft_consensus.cc:374] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:58.791244 16867 raft_consensus.cc:729] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088f9fb429d34b12b20ab1e6615787cb, State: Initialized, Role: FOLLOWER
I20260929 20:56:58.791086 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:58.760619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33027 (local address 127.0.0.1:32866)
0929 20:56:58.783211 (+ 22592us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:58.783294 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:58.783437 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:58.784516 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:58.784522 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:58.784549 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:58.784999 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:58.785015 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.786543 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.786549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:58.788486 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:58.788502 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.790225 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.790242 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:58.790405 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:58.790439 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:56:58.790538 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 20:56:58.790755 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22454,"thread_start_us":22279,"threads_started":1}
I20260929 20:56:58.791504 16867 consensus_queue.cc:260] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } }
I20260929 20:56:58.794068 16587 heartbeater.cc:503] Master 127.0.97.62:37141 was elected leader, sending a full tablet report...
I20260929 20:56:58.794094 16867 ts_tablet_manager.cc:1428] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20260929 20:56:58.795703 16657 tablet_service.cc:1467] Processing CreateTablet for tablet 4dab07b90aad478192c8c496ccaaabcb (DEFAULT_TABLE table=test-table [id=d4445629e2294e488b4dd8e609e0890b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:56:58.797020 16657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dab07b90aad478192c8c496ccaaabcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:56:58.804663 16870 tablet_bootstrap.cc:492] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135: Bootstrap starting.
I20260929 20:56:58.808423 16870 tablet_bootstrap.cc:654] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:58.809746 16870 log.cc:824] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:58.810700 16871 tablet_bootstrap.cc:492] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de: Bootstrap starting.
I20260929 20:56:58.813037 16870 tablet_bootstrap.cc:492] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135: No bootstrap required, opened a new log
I20260929 20:56:58.813279 16870 ts_tablet_manager.cc:1397] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260929 20:56:58.813933 16871 tablet_bootstrap.cc:654] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de: Neither blocks nor log segments found. Creating new log.
I20260929 20:56:58.815018 16871 log.cc:824] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de: Log is configured to *not* fsync() on all Append() calls
I20260929 20:56:58.817587 16871 tablet_bootstrap.cc:492] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de: No bootstrap required, opened a new log
I20260929 20:56:58.817821 16871 ts_tablet_manager.cc:1397] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 20:56:58.820748 16870 raft_consensus.cc:348] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } }
I20260929 20:56:58.821213 16870 raft_consensus.cc:374] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:58.821328 16870 raft_consensus.cc:729] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8863f58ee6048fa991666f283747135, State: Initialized, Role: FOLLOWER
I20260929 20:56:58.821902 16870 consensus_queue.cc:260] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } }
I20260929 20:56:58.822989 16853 heartbeater.cc:503] Master 127.0.97.62:37141 was elected leader, sending a full tablet report...
I20260929 20:56:58.823545 16870 ts_tablet_manager.cc:1428] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20260929 20:56:58.825639 16871 raft_consensus.cc:348] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } }
I20260929 20:56:58.826042 16871 raft_consensus.cc:374] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:56:58.826128 16871 raft_consensus.cc:729] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05ff9f2ec4e48a6a3510f7de11fb9de, State: Initialized, Role: FOLLOWER
I20260929 20:56:58.826733 16871 consensus_queue.cc:260] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } }
I20260929 20:56:58.827898 16722 heartbeater.cc:503] Master 127.0.97.62:37141 was elected leader, sending a full tablet report...
I20260929 20:56:58.828519 16871 ts_tablet_manager.cc:1428] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
W20260929 20:56:58.936115 16854 tablet.cc:2369] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:56:58.973537 16723 tablet.cc:2369] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:56:58.986287 16869 raft_consensus.cc:482] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:56:58.986603 16869 raft_consensus.cc:504] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } }
I20260929 20:56:58.988232 16869 leader_election.cc:288] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8863f58ee6048fa991666f283747135 (127.0.97.3:34559), a05ff9f2ec4e48a6a3510f7de11fb9de (127.0.97.2:33027)
I20260929 20:56:58.994505 16865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:58.988697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42989 (local address 127.0.97.3:34559)
0929 20:56:58.988900 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:58.988906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:58.989497 (+ 591us) server_negotiation.cc:407] Connection header received
0929 20:56:58.989663 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:58.989667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:58.989728 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:58.989802 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:58.990368 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.991103 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.992829 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.993620 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.993735 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:58.993836 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:58.994107 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 20:56:58.994260 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:56:58.994577 16866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:58.988720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36131 (local address 127.0.97.2:33027)
0929 20:56:58.988924 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:58.988931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:58.989554 (+ 623us) server_negotiation.cc:407] Connection header received
0929 20:56:58.989704 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:58.989709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:58.989775 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:58.989845 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:56:58.990499 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.991183 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.992879 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.993665 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.993798 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:58.993893 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:58.994171 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 20:56:58.994331 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:56:58.994711 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:58.988578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34559 (local address 127.0.97.1:42989)
0929 20:56:58.989386 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:58.989401 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:58.989540 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:58.989829 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:58.989833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:58.989856 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:58.990211 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:58.990228 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.991241 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.991246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:58.992677 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:58.992691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.993796 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.993802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:58.993957 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:58.993985 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 20:56:58.994067 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 20:56:58.994249 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":1,"thread_start_us":503,"threads_started":1}
I20260929 20:56:58.995082 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:58.988637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33027 (local address 127.0.97.1:36131)
0929 20:56:58.989445 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:58.989460 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:56:58.989581 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:56:58.989874 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:56:58.989878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:56:58.989899 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:56:58.990344 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:58.990355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.991326 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.991331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:58.992705 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:56:58.992719 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:56:58.993859 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:56:58.993865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:56:58.994017 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:58.994040 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:56:58.994129 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 20:56:58.994291 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":2,"thread_start_us":510,"threads_started":1}
I20260929 20:56:58.995743 16808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dab07b90aad478192c8c496ccaaabcb" candidate_uuid: "088f9fb429d34b12b20ab1e6615787cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8863f58ee6048fa991666f283747135" is_pre_election: true
I20260929 20:56:58.996055 16677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dab07b90aad478192c8c496ccaaabcb" candidate_uuid: "088f9fb429d34b12b20ab1e6615787cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" is_pre_election: true
I20260929 20:56:58.996389 16808 raft_consensus.cc:2454] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088f9fb429d34b12b20ab1e6615787cb in term 0.
I20260929 20:56:58.996601 16677 raft_consensus.cc:2454] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088f9fb429d34b12b20ab1e6615787cb in term 0.
I20260929 20:56:58.997155 16475 leader_election.cc:302] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088f9fb429d34b12b20ab1e6615787cb, c8863f58ee6048fa991666f283747135; no voters:
I20260929 20:56:58.997632 16869 raft_consensus.cc:2781] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:56:58.997802 16869 raft_consensus.cc:482] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:56:58.997915 16869 raft_consensus.cc:3037] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:59.000263 16869 raft_consensus.cc:504] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } }
I20260929 20:56:59.000864 16869 leader_election.cc:288] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [CANDIDATE]: Term 1 election: Requested vote from peers c8863f58ee6048fa991666f283747135 (127.0.97.3:34559), a05ff9f2ec4e48a6a3510f7de11fb9de (127.0.97.2:33027)
I20260929 20:56:59.001434 16808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dab07b90aad478192c8c496ccaaabcb" candidate_uuid: "088f9fb429d34b12b20ab1e6615787cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8863f58ee6048fa991666f283747135"
I20260929 20:56:59.001494 16677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dab07b90aad478192c8c496ccaaabcb" candidate_uuid: "088f9fb429d34b12b20ab1e6615787cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de"
I20260929 20:56:59.001631 16808 raft_consensus.cc:3037] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:59.001706 16677 raft_consensus.cc:3037] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:56:59.003891 16808 raft_consensus.cc:2454] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088f9fb429d34b12b20ab1e6615787cb in term 1.
I20260929 20:56:59.003971 16677 raft_consensus.cc:2454] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088f9fb429d34b12b20ab1e6615787cb in term 1.
I20260929 20:56:59.004292 16475 leader_election.cc:302] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088f9fb429d34b12b20ab1e6615787cb, c8863f58ee6048fa991666f283747135; no voters:
I20260929 20:56:59.004645 16869 raft_consensus.cc:2781] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:56:59.005101 16869 raft_consensus.cc:686] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [term 1 LEADER]: Becoming Leader. State: Replica: 088f9fb429d34b12b20ab1e6615787cb, State: Running, Role: LEADER
I20260929 20:56:59.005511 16869 consensus_queue.cc:237] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } } peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } }
I20260929 20:56:59.009651 16399 catalog_manager.cc:5495] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 088f9fb429d34b12b20ab1e6615787cb (127.0.97.1). New cstate: current_term: 1 leader_uuid: "088f9fb429d34b12b20ab1e6615787cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088f9fb429d34b12b20ab1e6615787cb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45865 } health_report { overall_health: HEALTHY } } }
W20260929 20:56:59.011327 16588 tablet.cc:2369] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:56:59.030412 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:59.024538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52272 (local address 127.0.97.1:45865)
0929 20:56:59.025179 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:59.025183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:59.025202 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:56:59.025286 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:59.025290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:59.025379 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:59.025506 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:56:59.025923 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:59.026792 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:59.028183 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:59.028631 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:59.028729 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:59.029136 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:56:59.029736 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:56:59.029851 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:59.029956 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 20:56:59.030024 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":387,"threads_started":1}
I20260929 20:56:59.036366 16677 raft_consensus.cc:1261] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de [term 1 FOLLOWER]: Refusing update from remote peer 088f9fb429d34b12b20ab1e6615787cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:56:59.036387 16808 raft_consensus.cc:1261] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135 [term 1 FOLLOWER]: Refusing update from remote peer 088f9fb429d34b12b20ab1e6615787cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:56:59.037357 16878 consensus_queue.cc:1035] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8863f58ee6048fa991666f283747135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:56:59.037711 16869 consensus_queue.cc:1035] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:56:59.057893 16883 mvcc.cc:205] Tried to move back new op lower bound from 7334770356362776576 to 7334770356249014272. Current Snapshot: MvccSnapshot[applied={T|T < 7334770356362776576}]
I20260929 20:56:59.064090 16398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:56:59.064446 16398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:56:59.068564 16398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245: Sending DeleteTablet for 3 replicas of tablet 4dab07b90aad478192c8c496ccaaabcb
I20260929 20:56:59.069787 16521 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dab07b90aad478192c8c496ccaaabcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:56:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52266
I20260929 20:56:59.069820 16657 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dab07b90aad478192c8c496ccaaabcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:56:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32866
I20260929 20:56:59.069904 16788 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dab07b90aad478192c8c496ccaaabcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:56:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37144
I20260929 20:56:59.070735 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 16436
I20260929 20:56:59.085317 16891 tablet_replica.cc:330] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb: stopping tablet replica
I20260929 20:56:59.085582 16891 raft_consensus.cc:2229] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:56:59.086305 16891 raft_consensus.cc:2258] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:56:59.087803 16891 ts_tablet_manager.cc:1905] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:56:59.092722 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:59.072018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52284 (local address 127.0.97.1:45865)
0929 20:56:59.072207 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:59.072213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:59.072234 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:56:59.072306 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:59.072311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:59.072398 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:59.072510 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:59.074748 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:59.076001 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:59.078115 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:59.078956 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:59.079092 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:59.082989 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:59.083010 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:59.083032 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:59.083084 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:59.087807 (+ 4723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:59.089023 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:59.089035 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:59.089039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:59.089246 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:59.089732 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:59.089742 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:59.089745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:59.090335 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:59.090646 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:59.091973 (+ 1327us) server_negotiation.cc:299] Negotiation successful
0929 20:56:59.092264 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:56:59.093848 16582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:59.094254 16892 tablet_replica.cc:330] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de: stopping tablet replica
I20260929 20:56:59.094813 16892 raft_consensus.cc:2229] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:56:59.095433 16892 raft_consensus.cc:2258] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:56:59.096263 16893 tablet_replica.cc:330] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135: stopping tablet replica
I20260929 20:56:59.096662 16893 raft_consensus.cc:2229] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:56:59.097069 16893 raft_consensus.cc:2258] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:56:59.097744 16892 ts_tablet_manager.cc:1905] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:56:59.099197 16893 ts_tablet_manager.cc:1905] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:56:59.103945 16892 ts_tablet_manager.cc:1918] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:56:59.104100 16892 log.cc:1196] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/4dab07b90aad478192c8c496ccaaabcb
I20260929 20:56:59.104511 16892 ts_tablet_manager.cc:1939] T 4dab07b90aad478192c8c496ccaaabcb P a05ff9f2ec4e48a6a3510f7de11fb9de: Deleting consensus metadata
I20260929 20:56:59.105417 16893 ts_tablet_manager.cc:1918] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:56:59.105572 16893 log.cc:1196] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/4dab07b90aad478192c8c496ccaaabcb
I20260929 20:56:59.106063 16893 ts_tablet_manager.cc:1939] T 4dab07b90aad478192c8c496ccaaabcb P c8863f58ee6048fa991666f283747135: Deleting consensus metadata
I20260929 20:56:59.106125 16385 catalog_manager.cc:4841] TS a05ff9f2ec4e48a6a3510f7de11fb9de (127.0.97.2:33027): tablet 4dab07b90aad478192c8c496ccaaabcb (table test-table [id=d4445629e2294e488b4dd8e609e0890b]) successfully deleted
I20260929 20:56:59.107839 16386 catalog_manager.cc:4841] TS c8863f58ee6048fa991666f283747135 (127.0.97.3:34559): tablet 4dab07b90aad478192c8c496ccaaabcb (table test-table [id=d4445629e2294e488b4dd8e609e0890b]) successfully deleted
I20260929 20:56:59.195712 16891 ts_tablet_manager.cc:1918] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:56:59.195935 16891 log.cc:1196] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/4dab07b90aad478192c8c496ccaaabcb
I20260929 20:56:59.196463 16891 ts_tablet_manager.cc:1939] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb: Deleting consensus metadata
I20260929 20:56:59.196491 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 16436
W20260929 20:56:59.211916 16386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45865 (error 108)
I20260929 20:56:59.212389 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 16591
I20260929 20:56:59.213619 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:59.213141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45865 (local address 127.0.0.1:52298)
0929 20:56:59.213397 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:59.213487 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260929 20:56:59.213866 16386 catalog_manager.cc:4584] TS 088f9fb429d34b12b20ab1e6615787cb (127.0.97.1:45865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4dab07b90aad478192c8c496ccaaabcb: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45865: connect: Connection refused (error 111)
I20260929 20:56:59.229840 16866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:59.213134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32874 (local address 127.0.97.2:33027)
0929 20:56:59.213315 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:59.213321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:59.213351 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:56:59.213527 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:59.213533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:59.213604 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:59.213684 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:59.215062 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:59.216005 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:59.217348 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:59.217761 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:59.217877 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:59.224115 (+ 6238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:59.224128 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:59.224142 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:59.224178 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:59.227026 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:59.227772 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:59.227779 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:59.227782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:59.227922 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:59.228288 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:59.228294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:59.228298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:59.228706 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:59.228974 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:59.229281 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 20:56:59.229497 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:56:59.230535 16717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:59.267495 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:59.266956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45865 (local address 127.0.0.1:52306)
0929 20:56:59.267185 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:59.267313 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 20:56:59.268642 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:59.268226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45865 (local address 127.0.0.1:52310)
0929 20:56:59.268383 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:59.268469 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 20:56:59.320330 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 16591
I20260929 20:56:59.329320 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:59.328880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45865 (local address 127.0.0.1:52320)
0929 20:56:59.329079 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:59.329169 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 20:56:59.330346 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:59.329963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45865 (local address 127.0.0.1:52332)
0929 20:56:59.330114 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:59.330229 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 20:56:59.334586 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 16725
I20260929 20:56:59.348052 16865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:59.335363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37148 (local address 127.0.97.3:34559)
0929 20:56:59.335539 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:56:59.335543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:56:59.335568 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:56:59.335672 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:56:59.335676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:56:59.335761 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:56:59.335849 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:56:59.337131 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:59.337895 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:59.339291 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:56:59.339744 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:56:59.339853 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:56:59.342734 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:56:59.342750 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:56:59.342767 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:56:59.342809 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:56:59.345304 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:59.346004 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:59.346013 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:59.346015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:59.346232 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:56:59.346595 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:56:59.346599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:56:59.346601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:56:59.346992 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:56:59.347228 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 20:56:59.347589 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 20:56:59.347784 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:56:59.348718 16848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:59.410878 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:59.410408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45865 (local address 127.0.0.1:52346)
0929 20:56:59.410611 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:59.410699 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 20:56:59.411906 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:56:59.411527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45865 (local address 127.0.0.1:52362)
0929 20:56:59.411714 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 20:56:59.411796 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 20:56:59.436388 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 16725
I20260929 20:56:59.450064 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 16366
I20260929 20:56:59.451273 16429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:56:59.564982 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 16366
I20260929 20:57:19.583829 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37141
--webserver_interface=127.0.97.62
--webserver_port=38113
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:57:19.932865 16899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:19.933185 16899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:19.933303 16899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:19.940688 16899 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 20:57:19.940804 16899 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 20:57:19.940842 16899 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 20:57:19.940894 16899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:57:19.940964 16899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:19.941064 16899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:19.941114 16899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:57:19.941187 16899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:57:19.951347 16899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:19.952744 16899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:19.954156 16899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:19.966737 16905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:19.966830 16904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:19.966810 16907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:19.967547 16899 server_base.cc:1034] running on GCE node
I20260929 20:57:19.968436 16899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:19.970448 16899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:19.971717 16899 hybrid_clock.cc:648] HybridClock initialized: now 1790715439971608 us; error 59 us; skew 500 ppm
I20260929 20:57:19.972074 16899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:57:19 dist-test-slave-m9g1 krb5kdc[15814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:57:19.975323 16899 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:57:19.976948 16899 webserver.cc:460] Webserver started at http://127.0.97.62:38113/ using document root <none> and password file <none>
I20260929 20:57:19.977682 16899 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:19.977785 16899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:19.983579 16899 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:57:19.986717 16914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:19.988322 16899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:57:19.988519 16899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "f4707c4c1d364f9aa579a7f8c75be245"
format_stamp: "Formatted at 2026-09-29 20:56:34 on dist-test-slave-m9g1"
server_key: "f978a6b74d0b582be12154a001318628"
server_key_iv: "68e7bb4f2566df506303fc4036c367b2"
server_key_version: "encryptionkey@0"
I20260929 20:57:19.989634 16899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:20.008446 16899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:20.012817 16899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:20.013116 16899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:20.031826 16899 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37141
I20260929 20:57:20.031872 16965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37141 every 8 connection(s)
I20260929 20:57:20.033444 16899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:57:20.037563 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 16899
I20260929 20:57:20.038475 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45865
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37141
--webserver_port=43945
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:57:20.041108 16966 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:57:20.044831 16966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245: Bootstrap starting.
I20260929 20:57:20.075086 16966 log.cc:824] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:20.097847 16966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:57:20.098630 16966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245: Bootstrap complete.
I20260929 20:57:20.112185 16966 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4707c4c1d364f9aa579a7f8c75be245" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } }
I20260929 20:57:20.113181 16966 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4707c4c1d364f9aa579a7f8c75be245, State: Initialized, Role: FOLLOWER
I20260929 20:57:20.113870 16966 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4707c4c1d364f9aa579a7f8c75be245" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } }
I20260929 20:57:20.114110 16966 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:57:20.114265 16966 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:57:20.114437 16966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 2 FOLLOWER]: Advancing to term 3
I20260929 20:57:20.118495 16966 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4707c4c1d364f9aa579a7f8c75be245" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } }
I20260929 20:57:20.119210 16966 leader_election.cc:302] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4707c4c1d364f9aa579a7f8c75be245; no voters:
I20260929 20:57:20.119921 16966 leader_election.cc:288] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 20:57:20.120080 16970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 20:57:20.122867 16970 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [term 3 LEADER]: Becoming Leader. State: Replica: f4707c4c1d364f9aa579a7f8c75be245, State: Running, Role: LEADER
I20260929 20:57:20.123473 16970 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4707c4c1d364f9aa579a7f8c75be245" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } }
I20260929 20:57:20.124025 16966 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:57:20.126084 16972 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4707c4c1d364f9aa579a7f8c75be245. Latest consensus state: current_term: 3 leader_uuid: "f4707c4c1d364f9aa579a7f8c75be245" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4707c4c1d364f9aa579a7f8c75be245" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } }
I20260929 20:57:20.126569 16972 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [sys.catalog]: This master's current role is: LEADER
I20260929 20:57:20.127197 16971 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f4707c4c1d364f9aa579a7f8c75be245" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4707c4c1d364f9aa579a7f8c75be245" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } }
I20260929 20:57:20.127497 16971 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245 [sys.catalog]: This master's current role is: LEADER
I20260929 20:57:20.137379 16975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:57:20.146739 16975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:57:20.148458 16975 catalog_manager.cc:1259] Loaded cluster ID: ae20801ac44f41adaaef378b12656ebe
I20260929 20:57:20.148667 16975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:57:20.152851 16975 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:57:20.154590 16975 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245: Loaded TSK: 0
I20260929 20:57:20.154904 16975 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f4707c4c1d364f9aa579a7f8c75be245: Loaded TSK: 1
F20260929 20:57:20.156100 16975 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7c487929ad google::LogMessage::Fail() at ??:0
@ 0x7f7c4879588a google::LogMessage::SendToLog() at ??:0
@ 0x7f7c487924de google::LogMessage::Flush() at ??:0
@ 0x7f7c48793a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7c4c4f67d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7c4c4e573e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7c5e1c11c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7c5e1c0783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7c5e24c2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f7c5e24c0c4 std::_Function_handler<>::_M_invoke() at ??:0
I20260929 20:57:20.190099 16988 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f7c5e2965e9 std::function<>::operator()() at ??:0
@ 0x7f7c5e1c4cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7c5e1c2c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7c5e24a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7c5e24a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7c4b3de345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7c4b3e191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7c4b3e1799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7c4b3bceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7c4e6966db start_thread at ??:0
@ 0x7f7c4762c61f clone at ??:0
W20260929 20:57:20.443140 16968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:20.443461 16968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:20.443611 16968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:20.451247 16968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:20.451390 16968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:20.451565 16968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:57:20.461746 16968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43945
--tserver_master_addrs=127.0.97.62:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:20.463124 16968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:20.464536 16968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:20.478089 17014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:20.478065 17016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:20.478235 17013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:20.478703 16968 server_base.cc:1034] running on GCE node
I20260929 20:57:20.479342 16968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:20.480340 16968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:20.481551 16968 hybrid_clock.cc:648] HybridClock initialized: now 1790715440481492 us; error 52 us; skew 500 ppm
I20260929 20:57:20.481876 16968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:57:20 dist-test-slave-m9g1 krb5kdc[15814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:57:20.484992 16968 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:57:20.486565 16968 webserver.cc:460] Webserver started at http://127.0.97.1:43945/ using document root <none> and password file <none>
I20260929 20:57:20.487254 16968 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:20.487375 16968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:20.492844 16968 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:57:20.495745 17023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:20.497243 16968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:57:20.497473 16968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "088f9fb429d34b12b20ab1e6615787cb"
format_stamp: "Formatted at 2026-09-29 20:56:34 on dist-test-slave-m9g1"
server_key: "fc00edefbfa8aaa12f6e0b9eff507e7c"
server_key_iv: "4f41073b8b11f95d76740283b051a87c"
server_key_version: "encryptionkey@0"
I20260929 20:57:20.498579 16968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:20.516453 16968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:20.521219 16968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:20.521481 16968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:20.522573 16968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:20.524552 17030 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260929 20:57:20.530344 17030 ts_tablet_manager.cc:1852] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 20:57:20.531437 17030 ts_tablet_manager.cc:1905] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:57:20.539299 17030 ts_tablet_manager.cc:1918] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:57:20.539572 17030 ts_tablet_manager.cc:1939] T 4dab07b90aad478192c8c496ccaaabcb P 088f9fb429d34b12b20ab1e6615787cb: Deleting consensus metadata
I20260929 20:57:20.540047 16968 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 20:57:20.540184 16968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.000s sys 0.001s
I20260929 20:57:20.540324 16968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:20.540397 16968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:20.574609 16968 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45865
I20260929 20:57:20.574669 17136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45865 every 8 connection(s)
I20260929 20:57:20.576370 16968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:57:20.577839 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 16968
I20260929 20:57:20.578715 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33027
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37141
--webserver_port=33685
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:57:20.582854 17139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:20.580673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:59473)
0929 20:57:20.581567 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:20.581826 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":400,"threads_started":1}
W20260929 20:57:20.583627 17137 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37141 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37141: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
I20260929 20:57:20.584635 17139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:20.584206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:49627)
0929 20:57:20.584396 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:20.584477 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 20:57:20.585551 17139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:20.585188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:56415)
0929 20:57:20.585338 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:20.585415 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 20:57:20.585775 17137 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:57:20.918787 17140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:20.919091 17140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:20.919200 17140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:20.926644 17140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:20.926771 17140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:20.926887 17140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:57:20.936776 17140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33685
--tserver_master_addrs=127.0.97.62:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:20.938107 17140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:20.939465 17140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:20.952736 17148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:20.952705 17146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:20.953194 17145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:20.953641 17140 server_base.cc:1034] running on GCE node
I20260929 20:57:20.954372 17140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:20.955430 17140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:20.956668 17140 hybrid_clock.cc:648] HybridClock initialized: now 1790715440956623 us; error 44 us; skew 500 ppm
I20260929 20:57:20.957020 17140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:57:20 dist-test-slave-m9g1 krb5kdc[15814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:57:20.960608 17140 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:57:20.962265 17140 webserver.cc:460] Webserver started at http://127.0.97.2:33685/ using document root <none> and password file <none>
I20260929 20:57:20.963028 17140 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:20.963157 17140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:20.968953 17140 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:57:20.972160 17155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:20.973692 17140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:57:20.973858 17140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "a05ff9f2ec4e48a6a3510f7de11fb9de"
format_stamp: "Formatted at 2026-09-29 20:56:35 on dist-test-slave-m9g1"
server_key: "4d7c59596e5058aaa4586c868e60b4dc"
server_key_iv: "83e0f028a35d9ed4125921ea6779368a"
server_key_version: "encryptionkey@0"
I20260929 20:57:20.974946 17140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:20.991269 17140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:20.995692 17140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:20.995935 17140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:20.996948 17140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:20.998431 17140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:20.998549 17140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:20.998661 17140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:20.998734 17140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:21.030380 17140 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33027
I20260929 20:57:21.030448 17267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33027 every 8 connection(s)
I20260929 20:57:21.032018 17140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:57:21.033764 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 17140
I20260929 20:57:21.034420 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34559
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37141
--webserver_port=41287
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:57:21.040468 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:21.035398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:41751)
0929 20:57:21.038950 (+ 3552us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:21.039250 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3201,"thread_start_us":2946,"threads_started":1}
W20260929 20:57:21.041265 17268 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37141 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37141: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
I20260929 20:57:21.042335 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:21.041868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:47653)
0929 20:57:21.042080 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:21.042166 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 20:57:21.043361 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:21.042950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:33965)
0929 20:57:21.043130 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:21.043226 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260929 20:57:21.043663 17268 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:57:21.367288 17271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:21.367594 17271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:21.367722 17271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:21.375291 17271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:21.375414 17271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:21.375571 17271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:57:21.385571 17271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41287
--tserver_master_addrs=127.0.97.62:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:21.386967 17271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:21.388294 17271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:21.402307 17277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:21.402475 17279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:21.402830 17276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:21.403146 17271 server_base.cc:1034] running on GCE node
I20260929 20:57:21.403801 17271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:21.404747 17271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:21.405943 17271 hybrid_clock.cc:648] HybridClock initialized: now 1790715441405891 us; error 41 us; skew 500 ppm
I20260929 20:57:21.406320 17271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:57:21 dist-test-slave-m9g1 krb5kdc[15814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:57:21.409416 17271 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:57:21.410907 17271 webserver.cc:460] Webserver started at http://127.0.97.3:41287/ using document root <none> and password file <none>
I20260929 20:57:21.411599 17271 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:21.411713 17271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:21.417244 17271 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:57:21.420580 17286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:21.422277 17271 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 20:57:21.422457 17271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "c8863f58ee6048fa991666f283747135"
format_stamp: "Formatted at 2026-09-29 20:56:35 on dist-test-slave-m9g1"
server_key: "bf8df371bbd5bbbcc0752f822991dae7"
server_key_iv: "0f72bf91c6922dc9015b0715c9588716"
server_key_version: "encryptionkey@0"
I20260929 20:57:21.423482 17271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:21.455892 17271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:21.460523 17271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:21.460800 17271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:21.461810 17271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:21.463255 17271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:21.463339 17271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:21.463438 17271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:21.463500 17271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:21.493970 17271 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34559
I20260929 20:57:21.494031 17398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34559 every 8 connection(s)
I20260929 20:57:21.495422 17271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:57:21.499254 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 17271
I20260929 20:57:21.500270 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:21.498417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:34575)
0929 20:57:21.499135 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:21.499329 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":267,"threads_started":1}
W20260929 20:57:21.500869 17399 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37141 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37141: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
I20260929 20:57:21.501654 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:21.501361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:47037)
0929 20:57:21.501480 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:21.501554 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 20:57:21.502498 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:21.502161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:34757)
0929 20:57:21.502312 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:21.502388 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 20:57:21.502719 17399 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 20:57:21.587306 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:21.586453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:40695)
0929 20:57:21.586974 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:21.587069 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":247,"threads_started":1}
I20260929 20:57:22.045212 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:22.044297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:56011)
0929 20:57:22.044853 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:22.044951 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":284,"threads_started":1}
I20260929 20:57:22.504190 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:22.503335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:47823)
0929 20:57:22.503834 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:22.503948 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":251,"threads_started":1}
I20260929 20:57:22.588876 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:22.588208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:35947)
0929 20:57:22.588614 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:22.588701 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":213,"threads_started":1}
I20260929 20:57:23.047263 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:23.046412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:58355)
0929 20:57:23.046980 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:23.047084 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":305,"threads_started":1}
I20260929 20:57:23.505836 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:23.505094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:46623)
0929 20:57:23.505568 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:23.505665 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":228,"threads_started":1}
I20260929 20:57:23.590644 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:23.589865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:58785)
0929 20:57:23.590383 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:23.590477 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":219,"threads_started":1}
I20260929 20:57:24.049140 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:24.048305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:44421)
0929 20:57:24.048857 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:24.048950 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":315,"threads_started":1}
I20260929 20:57:24.507481 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:24.506766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:57231)
0929 20:57:24.507217 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:24.507305 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":222,"threads_started":1}
I20260929 20:57:24.592239 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:24.591527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:45443)
0929 20:57:24.591974 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:24.592058 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":218,"threads_started":1}
I20260929 20:57:25.051316 17413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:25.050375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:55579)
0929 20:57:25.051014 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:25.051099 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":291,"threads_started":1}
I20260929 20:57:25.509281 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:25.508512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:34467)
0929 20:57:25.509002 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:25.509097 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":237,"threads_started":1}
I20260929 20:57:25.593890 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:25.593180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:36441)
0929 20:57:25.593645 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:25.593731 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":235,"threads_started":1}
I20260929 20:57:26.053303 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:26.052424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:36557)
0929 20:57:26.052995 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:26.053098 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":313,"threads_started":1}
I20260929 20:57:26.510974 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:26.510247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:35067)
0929 20:57:26.510713 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:26.510797 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":224,"threads_started":1}
I20260929 20:57:26.595606 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:26.594831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:40447)
0929 20:57:26.595351 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:26.595430 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":297,"threads_started":1}
I20260929 20:57:27.055195 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:27.054438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:58287)
0929 20:57:27.054897 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:27.054997 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":234,"threads_started":1}
I20260929 20:57:27.512620 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:27.511907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:38491)
0929 20:57:27.512359 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:27.512455 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":221,"threads_started":1}
I20260929 20:57:27.597406 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:27.596565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:48889)
0929 20:57:27.597102 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:27.597205 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":234,"threads_started":1}
I20260929 20:57:28.057189 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:28.056264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:35991)
0929 20:57:28.056794 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:28.056886 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":224,"threads_started":1}
I20260929 20:57:28.514503 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:28.513589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:47823)
0929 20:57:28.514129 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:28.514286 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":253,"threads_started":1}
I20260929 20:57:28.599282 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:28.598455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:50201)
0929 20:57:28.599001 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:28.599091 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":314,"threads_started":1}
I20260929 20:57:29.059221 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:29.058396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:57687)
0929 20:57:29.058893 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:29.058985 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":241,"threads_started":1}
I20260929 20:57:29.516240 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:29.515505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:38005)
0929 20:57:29.515982 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:29.516068 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":250,"threads_started":1}
I20260929 20:57:29.600890 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:29.600141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:37819)
0929 20:57:29.600626 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:29.600733 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":238,"threads_started":1}
I20260929 20:57:30.061280 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:30.060336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:40725)
0929 20:57:30.060925 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:30.061040 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":255,"threads_started":1}
I20260929 20:57:30.517970 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:30.517200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:51499)
0929 20:57:30.517695 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:30.517796 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":244,"threads_started":1}
I20260929 20:57:30.602662 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:30.601924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:56471)
0929 20:57:30.602398 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:30.602480 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":262,"threads_started":1}
I20260929 20:57:31.063210 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:31.062437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:37615)
0929 20:57:31.062952 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:31.063039 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":279,"threads_started":1}
I20260929 20:57:31.519773 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:31.518995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:42189)
0929 20:57:31.519502 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:31.519593 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":288,"threads_started":1}
I20260929 20:57:31.604323 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:31.603567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:54145)
0929 20:57:31.604073 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:31.604156 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":300,"threads_started":1}
I20260929 20:57:32.065097 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:32.064305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:58479)
0929 20:57:32.064760 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:32.064855 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":227,"threads_started":1}
I20260929 20:57:32.521657 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:32.520822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:34793)
0929 20:57:32.521323 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:32.521427 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":221,"threads_started":1}
I20260929 20:57:32.606038 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:32.605305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:53925)
0929 20:57:32.605776 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:32.605863 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":222,"threads_started":1}
I20260929 20:57:33.066979 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:33.066245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:47125)
0929 20:57:33.066728 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:33.066817 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":229,"threads_started":1}
I20260929 20:57:33.523339 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:33.522615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:40755)
0929 20:57:33.523087 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:33.523175 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":238,"threads_started":1}
I20260929 20:57:33.607723 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:33.606945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:58857)
0929 20:57:33.607434 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:33.607552 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":245,"threads_started":1}
I20260929 20:57:34.068893 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:34.068029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:44881)
0929 20:57:34.068557 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:34.068651 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":275,"threads_started":1}
I20260929 20:57:34.525058 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:34.524283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:38283)
0929 20:57:34.524780 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:34.524871 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":252,"threads_started":1}
I20260929 20:57:34.609520 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:34.608730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:39601)
0929 20:57:34.609189 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:34.609305 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":225,"threads_started":1}
I20260929 20:57:35.070755 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:35.069885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:43523)
0929 20:57:35.070423 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:35.070519 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":245,"threads_started":1}
I20260929 20:57:35.526736 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:35.525992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:35989)
0929 20:57:35.526455 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:35.526536 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":258,"threads_started":1}
I20260929 20:57:35.611464 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:35.610746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.1:39941)
0929 20:57:35.611213 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:35.611298 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":234,"threads_started":1}
I20260929 20:57:36.072758 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:36.071932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.2:38975)
0929 20:57:36.072429 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:36.072528 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":266,"threads_started":1}
I20260929 20:57:36.501585 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 16968
I20260929 20:57:36.525602 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:36.502648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.0.97.1:45865)
0929 20:57:36.503349 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:36.503363 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:36.503405 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:57:36.505407 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:36.505420 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:36.505951 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:36.506319 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:36.507605 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:36.508902 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:36.510174 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:36.510907 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:36.511051 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:36.519777 (+ 8726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:36.519786 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:36.519804 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:36.519875 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:36.522748 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:36.523392 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:36.523398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:36.523405 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:36.523570 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:36.523885 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:36.523888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:36.523890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:36.524431 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:36.524655 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:36.524989 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 20:57:36.525312 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":379,"threads_started":1}
I20260929 20:57:36.527007 17132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:36.528515 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:36.527714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.97.3:40825)
0929 20:57:36.528193 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:36.528300 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":225,"threads_started":1}
I20260929 20:57:36.614809 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 16968
I20260929 20:57:36.628609 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 17140
I20260929 20:57:36.645485 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:36.629490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43678 (local address 127.0.97.2:33027)
0929 20:57:36.630034 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:36.630049 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:36.630085 (+ 36us) server_negotiation.cc:407] Connection header received
0929 20:57:36.631681 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:36.631690 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:36.632110 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:36.632461 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:36.633845 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:36.635173 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:36.636392 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:36.637042 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:36.637209 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:36.639622 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:36.639630 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:36.639646 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:36.639689 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:36.642395 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:36.643045 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:36.643053 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:36.643059 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:36.643225 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:36.643575 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:36.643580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:36.643583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:36.644089 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:36.644389 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:36.644798 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 20:57:36.645192 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":255,"threads_started":1}
I20260929 20:57:36.647018 17263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:36.730600 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 17140
I20260929 20:57:36.745553 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 17271
I20260929 20:57:36.762460 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:36.746594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34732 (local address 127.0.97.3:34559)
0929 20:57:36.747277 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:36.747298 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:36.747341 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:57:36.748934 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:36.748945 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:36.749385 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:36.749669 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:36.750992 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:36.752270 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:36.753409 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:36.754048 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:36.754174 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:36.756647 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:36.756655 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:36.756670 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:36.756720 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:36.759390 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:36.760055 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:36.760063 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:36.760069 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:36.760264 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:36.760549 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:36.760552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:36.760554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:36.761054 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:36.761304 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:36.761698 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 20:57:36.762097 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":336,"threads_started":1}
I20260929 20:57:36.763779 17394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:36.846621 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 17271
2026-09-29T20:57:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64246 ms)
[----------] 30 tests from SecurityITest (287370 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:57:36 dist-test-slave-m9g1 krb5kdc[17457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:57:36 dist-test-slave-m9g1 krb5kdc[17457](info): set up 2 sockets
Sep 29 20:57:36 dist-test-slave-m9g1 krb5kdc[17457](info): commencing operation
krb5kdc: starting...
W20260929 20:57:37.858079 388 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.018s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:57:38 dist-test-slave-m9g1 krb5kdc[17457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:57:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:57:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:57:38.098341 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39347
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:57:38.418483 17470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:38.418820 17470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:38.418956 17470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:38.426160 17470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:57:38.426311 17470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:38.426385 17470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:38.426465 17470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:57:38.426527 17470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:57:38.436851 17470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:38.438277 17470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:38.439596 17470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:38.451362 17475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:38.451347 17476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:38.451424 17478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:38.452080 17470 server_base.cc:1034] running on GCE node
I20260929 20:57:38.452864 17470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:38.454388 17470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:38.455610 17470 hybrid_clock.cc:648] HybridClock initialized: now 1790715458455556 us; error 38 us; skew 500 ppm
I20260929 20:57:38.456001 17470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:57:38 dist-test-slave-m9g1 krb5kdc[17457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:57:38.458992 17470 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:57:38.460498 17470 webserver.cc:460] Webserver started at http://127.0.97.62:42785/ using document root <none> and password file <none>
I20260929 20:57:38.461238 17470 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:38.461341 17470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:38.461688 17470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:38.464763 17470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "c23d88eee5314f33bb79aca5e2f7bd99"
format_stamp: "Formatted at 2026-09-29 20:57:38 on dist-test-slave-m9g1"
server_key: "684e61f64faf7b108af5358bc6298c4d"
server_key_iv: "0c1279660db22ba6330d180adb9677d2"
server_key_version: "encryptionkey@0"
I20260929 20:57:38.465515 17470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "c23d88eee5314f33bb79aca5e2f7bd99"
format_stamp: "Formatted at 2026-09-29 20:57:38 on dist-test-slave-m9g1"
server_key: "684e61f64faf7b108af5358bc6298c4d"
server_key_iv: "0c1279660db22ba6330d180adb9677d2"
server_key_version: "encryptionkey@0"
I20260929 20:57:38.470907 17470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 20:57:38.474571 17485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:38.476119 17470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:57:38.476311 17470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "c23d88eee5314f33bb79aca5e2f7bd99"
format_stamp: "Formatted at 2026-09-29 20:57:38 on dist-test-slave-m9g1"
server_key: "684e61f64faf7b108af5358bc6298c4d"
server_key_iv: "0c1279660db22ba6330d180adb9677d2"
server_key_version: "encryptionkey@0"
I20260929 20:57:38.476516 17470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:38.507839 17470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:38.512372 17470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:38.512646 17470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:38.529577 17470 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39347
I20260929 20:57:38.529605 17536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39347 every 8 connection(s)
I20260929 20:57:38.530947 17470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:57:38.534643 17537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:38.540678 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 17470
I20260929 20:57:38.540875 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:57:38.541481 388 external_mini_cluster.cc:1428] Setting key 42644bdc6585513aa0df1fa1ec03a667
I20260929 20:57:38.544771 17537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99: Bootstrap starting.
I20260929 20:57:38.548214 17537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:38.549325 17537 log.cc:824] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:38.552349 17537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99: No bootstrap required, opened a new log
Sep 29 20:57:38 dist-test-slave-m9g1 krb5kdc[17457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:57:38.560657 17537 raft_consensus.cc:348] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23d88eee5314f33bb79aca5e2f7bd99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39347 } }
I20260929 20:57:38.561100 17537 raft_consensus.cc:374] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:38.561215 17537 raft_consensus.cc:729] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c23d88eee5314f33bb79aca5e2f7bd99, State: Initialized, Role: FOLLOWER
I20260929 20:57:38.561829 17537 consensus_queue.cc:260] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23d88eee5314f33bb79aca5e2f7bd99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39347 } }
I20260929 20:57:38.562079 17537 raft_consensus.cc:388] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:57:38.562294 17537 raft_consensus.cc:482] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:57:38.562502 17537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:38.564186 17540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:38.543423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41170 (local address 127.0.97.62:39347)
0929 20:57:38.544156 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:38.544167 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:38.544215 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:57:38.545904 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:38.545929 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:38.546378 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:38.546707 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:38.548529 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:38.549897 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:38.551416 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:38.552419 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:38.552588 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:38.556733 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:38.556741 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:38.556756 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:38.556805 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:38.559855 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:38.560655 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:38.560662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:38.560668 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:38.560862 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:38.561287 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:38.561292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:38.561296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:38.561938 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:38.562247 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:38.562695 (+ 448us) server_negotiation.cc:299] Negotiation successful
0929 20:57:38.563080 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":313,"threads_started":1}
I20260929 20:57:38.565016 17537 raft_consensus.cc:504] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23d88eee5314f33bb79aca5e2f7bd99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39347 } }
I20260929 20:57:38.565544 17537 leader_election.cc:302] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c23d88eee5314f33bb79aca5e2f7bd99; no voters:
I20260929 20:57:38.566181 17537 leader_election.cc:288] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:57:38.566324 17542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:57:38.567624 17542 raft_consensus.cc:686] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99 [term 1 LEADER]: Becoming Leader. State: Replica: c23d88eee5314f33bb79aca5e2f7bd99, State: Running, Role: LEADER
I20260929 20:57:38.568152 17537 sys_catalog.cc:564] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:57:38.568084 17542 consensus_queue.cc:237] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23d88eee5314f33bb79aca5e2f7bd99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39347 } }
I20260929 20:57:38.571728 17543 sys_catalog.cc:455] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c23d88eee5314f33bb79aca5e2f7bd99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23d88eee5314f33bb79aca5e2f7bd99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39347 } } }
I20260929 20:57:38.571797 17544 sys_catalog.cc:455] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c23d88eee5314f33bb79aca5e2f7bd99. Latest consensus state: current_term: 1 leader_uuid: "c23d88eee5314f33bb79aca5e2f7bd99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c23d88eee5314f33bb79aca5e2f7bd99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39347 } } }
I20260929 20:57:38.572125 17543 sys_catalog.cc:458] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99 [sys.catalog]: This master's current role is: LEADER
I20260929 20:57:38.572170 17544 sys_catalog.cc:458] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99 [sys.catalog]: This master's current role is: LEADER
I20260929 20:57:38.572964 17551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:57:38.578962 17551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:57:38.595774 17551 catalog_manager.cc:1347] Generated new cluster ID: a1050f6fce5e478cb268256d88f9bc2a
I20260929 20:57:38.595916 17551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:57:38.608422 17551 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:57:38.610176 17551 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:57:38.618027 17551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99: Generated new TSK 0
I20260929 20:57:38.618966 17551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:57:38.677919 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39347
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:57:39.017639 17563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:39.018002 17563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:39.018151 17563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:39.025391 17563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:39.025511 17563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:39.025666 17563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:57:39.035565 17563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:39.036875 17563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:39.038210 17563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:39.051222 17569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:39.051337 17568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:39.051246 17571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:39.052187 17563 server_base.cc:1034] running on GCE node
I20260929 20:57:39.052800 17563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:39.053705 17563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:39.054921 17563 hybrid_clock.cc:648] HybridClock initialized: now 1790715459054851 us; error 36 us; skew 500 ppm
I20260929 20:57:39.055228 17563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:57:39 dist-test-slave-m9g1 krb5kdc[17457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:57:39.058393 17563 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:57:39.059885 17563 webserver.cc:460] Webserver started at http://127.0.97.1:45685/ using document root <none> and password file <none>
I20260929 20:57:39.060627 17563 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:39.060760 17563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:39.061055 17563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:39.064072 17563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "0738387bc0e84a808cbc507528df9199"
format_stamp: "Formatted at 2026-09-29 20:57:39 on dist-test-slave-m9g1"
server_key: "938b52d02fbcc9ac224e60c3727dce94"
server_key_iv: "bac2f523e0b29498d67acf47b1462a87"
server_key_version: "encryptionkey@0"
I20260929 20:57:39.064965 17563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0738387bc0e84a808cbc507528df9199"
format_stamp: "Formatted at 2026-09-29 20:57:39 on dist-test-slave-m9g1"
server_key: "938b52d02fbcc9ac224e60c3727dce94"
server_key_iv: "bac2f523e0b29498d67acf47b1462a87"
server_key_version: "encryptionkey@0"
I20260929 20:57:39.070472 17563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:57:39.074143 17578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:39.075606 17563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:57:39.075768 17563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "0738387bc0e84a808cbc507528df9199"
format_stamp: "Formatted at 2026-09-29 20:57:39 on dist-test-slave-m9g1"
server_key: "938b52d02fbcc9ac224e60c3727dce94"
server_key_iv: "bac2f523e0b29498d67acf47b1462a87"
server_key_version: "encryptionkey@0"
I20260929 20:57:39.075969 17563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:39.089183 17563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:39.093429 17563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:39.093703 17563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:39.094702 17563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:39.096035 17563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:39.096112 17563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:39.096226 17563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:39.096277 17563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:39.126971 17563 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46773
I20260929 20:57:39.127012 17690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46773 every 8 connection(s)
I20260929 20:57:39.128207 17563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:57:39.136772 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 17563
I20260929 20:57:39.136963 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:57:39.137558 388 external_mini_cluster.cc:1428] Setting key b9a178fa0596e38608644ae95857e4be
Sep 29 20:57:39 dist-test-slave-m9g1 krb5kdc[17457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:57:39.151081 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:39.130651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55973 (local address 127.0.97.62:39347)
0929 20:57:39.131107 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:39.131112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:39.131980 (+ 868us) server_negotiation.cc:407] Connection header received
0929 20:57:39.133799 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:39.133805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:39.133884 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:39.133979 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:39.135570 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:39.136278 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:39.137873 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:39.138433 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:39.138543 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:39.145136 (+ 6593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:39.145142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:39.145145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:39.145180 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:39.147930 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:39.148719 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:39.148724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:39.148728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:39.148870 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:39.149290 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:39.149294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:39.149297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:39.149631 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:39.149769 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:39.150527 (+ 758us) server_negotiation.cc:299] Negotiation successful
0929 20:57:39.150748 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":226,"threads_started":1}
I20260929 20:57:39.151983 17694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:39.131074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39347 (local address 127.0.97.1:55973)
0929 20:57:39.131814 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:39.131857 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:39.133512 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:39.134263 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:39.134271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:39.134808 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:57:39.135392 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:39.135416 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:39.136429 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:39.136433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:39.137729 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:39.137742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:39.138642 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:39.138650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:39.138882 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:39.142373 (+ 3491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:57:39.142400 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:57:39.144891 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:57:39.148106 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:57:39.148114 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:57:39.148128 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:57:39.148607 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:57:39.148978 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:57:39.148981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:57:39.148983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:57:39.149179 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:57:39.149747 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:57:39.149753 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:57:39.150091 (+ 338us) client_negotiation.cc:769] Sending connection context
0929 20:57:39.150417 (+ 326us) client_negotiation.cc:240] Negotiation successful
0929 20:57:39.150832 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":257,"threads_started":1}
I20260929 20:57:39.153571 17691 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39347
I20260929 20:57:39.153906 17691 heartbeater.cc:464] Registering TS with master...
I20260929 20:57:39.154649 17691 heartbeater.cc:511] Master 127.0.97.62:39347 requested a full tablet report, sending...
I20260929 20:57:39.157073 17502 ts_manager.cc:194] Registered new tserver with Master: 0738387bc0e84a808cbc507528df9199 (127.0.97.1:46773)
I20260929 20:57:39.159272 17502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55973
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:57:39.189458 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39347
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:57:39.504675 17697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:39.505103 17697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:39.505249 17697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:39.512598 17697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:39.512702 17697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:39.512851 17697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:57:39.522538 17697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:39.523806 17697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:39.525094 17697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:39.537931 17703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:39.537993 17702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:39.538179 17705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:39.538630 17697 server_base.cc:1034] running on GCE node
I20260929 20:57:39.539232 17697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:39.540155 17697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:39.541394 17697 hybrid_clock.cc:648] HybridClock initialized: now 1790715459541314 us; error 63 us; skew 500 ppm
I20260929 20:57:39.541709 17697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:57:39 dist-test-slave-m9g1 krb5kdc[17457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:57:39.544900 17697 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:57:39.546427 17697 webserver.cc:460] Webserver started at http://127.0.97.2:41959/ using document root <none> and password file <none>
I20260929 20:57:39.547134 17697 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:39.547250 17697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:39.547542 17697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:39.550441 17697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "6acd799d7a374bef938f93f08d82c52e"
format_stamp: "Formatted at 2026-09-29 20:57:39 on dist-test-slave-m9g1"
server_key: "1eb1a4cd261d55935456dc886ea5e833"
server_key_iv: "7c07c524a9d7d4ce0510efd7cf30e415"
server_key_version: "encryptionkey@0"
I20260929 20:57:39.551242 17697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6acd799d7a374bef938f93f08d82c52e"
format_stamp: "Formatted at 2026-09-29 20:57:39 on dist-test-slave-m9g1"
server_key: "1eb1a4cd261d55935456dc886ea5e833"
server_key_iv: "7c07c524a9d7d4ce0510efd7cf30e415"
server_key_version: "encryptionkey@0"
I20260929 20:57:39.556421 17697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:57:39.559897 17712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:39.561362 17697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:57:39.561542 17697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "6acd799d7a374bef938f93f08d82c52e"
format_stamp: "Formatted at 2026-09-29 20:57:39 on dist-test-slave-m9g1"
server_key: "1eb1a4cd261d55935456dc886ea5e833"
server_key_iv: "7c07c524a9d7d4ce0510efd7cf30e415"
server_key_version: "encryptionkey@0"
I20260929 20:57:39.561733 17697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:39.580159 17697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:39.584847 17697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:39.585134 17697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:39.586097 17697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:39.587463 17697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:39.587540 17697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:39.587662 17697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:39.587718 17697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:39.618114 17697 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36293
I20260929 20:57:39.618162 17824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36293 every 8 connection(s)
I20260929 20:57:39.619427 17697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:57:39.621327 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 17697
I20260929 20:57:39.621497 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:57:39.621975 388 external_mini_cluster.cc:1428] Setting key 349b8ee70c377fb97e7cf6a2448fc219
Sep 29 20:57:39 dist-test-slave-m9g1 krb5kdc[17457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:57:39.638747 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:39.622093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35953 (local address 127.0.97.62:39347)
0929 20:57:39.622334 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:39.622340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:39.623525 (+ 1185us) server_negotiation.cc:407] Connection header received
0929 20:57:39.625355 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:39.625360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:39.625440 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:39.625522 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:39.627058 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:39.627769 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:39.629296 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:39.629776 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:39.629884 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:39.633402 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:39.633407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:39.633410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:39.633442 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:39.635841 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:39.636529 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:39.636533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:39.636535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:39.636657 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:39.637054 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:39.637058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:39.637060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:39.637335 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:39.637461 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:39.638152 (+ 691us) server_negotiation.cc:299] Negotiation successful
0929 20:57:39.638422 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:57:39.639729 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:39.622579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39347 (local address 127.0.97.2:35953)
0929 20:57:39.623379 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:39.623429 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:39.625090 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:39.625767 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:39.625776 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:39.626327 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:57:39.626893 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:39.626911 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:39.627918 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:39.627922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:39.629145 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:39.629157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:39.629995 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:39.630002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:39.630257 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:39.630879 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:57:39.630905 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:57:39.633184 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:57:39.635968 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:57:39.635977 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:57:39.635994 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:57:39.636401 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:57:39.636761 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:57:39.636766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:57:39.636770 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:57:39.636947 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:57:39.637450 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:57:39.637457 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:57:39.637799 (+ 342us) client_negotiation.cc:769] Sending connection context
0929 20:57:39.638036 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 20:57:39.638601 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":318,"threads_started":1}
I20260929 20:57:39.641214 17825 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39347
I20260929 20:57:39.641546 17825 heartbeater.cc:464] Registering TS with master...
I20260929 20:57:39.642475 17825 heartbeater.cc:511] Master 127.0.97.62:39347 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:57:39.644119 17502 ts_manager.cc:194] Registered new tserver with Master: 6acd799d7a374bef938f93f08d82c52e (127.0.97.2:36293)
I20260929 20:57:39.645035 17502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:57:39.673269 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39347
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:57:40.009348 17830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:40.009641 17830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:40.009743 17830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:40.017055 17830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:40.017170 17830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:40.017340 17830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:57:40.027089 17830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:40.028344 17830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:40.029553 17830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:40.042563 17836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:40.042519 17838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:40.042588 17835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:40.043547 17830 server_base.cc:1034] running on GCE node
I20260929 20:57:40.044210 17830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:40.045270 17830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:40.046502 17830 hybrid_clock.cc:648] HybridClock initialized: now 1790715460046452 us; error 39 us; skew 500 ppm
I20260929 20:57:40.046878 17830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:57:40 dist-test-slave-m9g1 krb5kdc[17457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:57:40.050135 17830 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:57:40.051841 17830 webserver.cc:460] Webserver started at http://127.0.97.3:44585/ using document root <none> and password file <none>
I20260929 20:57:40.052564 17830 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:40.052675 17830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:40.052958 17830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:40.056142 17830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "cfc4d0be29d54acfb46e90ff584cff75"
format_stamp: "Formatted at 2026-09-29 20:57:40 on dist-test-slave-m9g1"
server_key: "33446c9240250e0cb017c689e1a00b47"
server_key_iv: "74fcb6d57344bbc95268a3b4b3ef8d44"
server_key_version: "encryptionkey@0"
I20260929 20:57:40.056936 17830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cfc4d0be29d54acfb46e90ff584cff75"
format_stamp: "Formatted at 2026-09-29 20:57:40 on dist-test-slave-m9g1"
server_key: "33446c9240250e0cb017c689e1a00b47"
server_key_iv: "74fcb6d57344bbc95268a3b4b3ef8d44"
server_key_version: "encryptionkey@0"
I20260929 20:57:40.062680 17830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:57:40.066969 17845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:40.068708 17830 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:57:40.068913 17830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "cfc4d0be29d54acfb46e90ff584cff75"
format_stamp: "Formatted at 2026-09-29 20:57:40 on dist-test-slave-m9g1"
server_key: "33446c9240250e0cb017c689e1a00b47"
server_key_iv: "74fcb6d57344bbc95268a3b4b3ef8d44"
server_key_version: "encryptionkey@0"
I20260929 20:57:40.069139 17830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:40.090912 17830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:40.095141 17830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:40.095413 17830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:40.096432 17830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:40.097777 17830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:40.097860 17830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:40.097970 17830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:40.098018 17830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:40.128259 17830 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36431
I20260929 20:57:40.128296 17957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36431 every 8 connection(s)
I20260929 20:57:40.129577 17830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:57:40.137048 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 17830
I20260929 20:57:40.137207 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:57:40.137660 388 external_mini_cluster.cc:1428] Setting key 196e46b86a0f24269a3deca3cb8a216d
Sep 29 20:57:40 dist-test-slave-m9g1 krb5kdc[17457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:57:40.150036 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:40.132427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52267 (local address 127.0.97.62:39347)
0929 20:57:40.132625 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:40.132631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:40.133895 (+ 1264us) server_negotiation.cc:407] Connection header received
0929 20:57:40.135815 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:40.135820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:40.135895 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:40.135985 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:40.137622 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.138526 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.140708 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.141125 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.141218 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:40.144658 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:40.144663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:40.144666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:40.144700 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:40.147203 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:40.147871 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:40.147877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:40.147880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:40.148003 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:40.148384 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:40.148389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:40.148393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:40.148660 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:40.148783 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:40.149479 (+ 696us) server_negotiation.cc:299] Negotiation successful
0929 20:57:40.149707 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:57:40.151006 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:40.132972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39347 (local address 127.0.97.3:52267)
0929 20:57:40.133724 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:40.133766 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:40.135536 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:40.136211 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:40.136219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:40.136756 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:57:40.137431 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:40.137452 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.138708 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.138713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:40.140552 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:40.140568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.141320 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.141326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:40.141523 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:40.142173 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:57:40.142230 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:57:40.144413 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:57:40.147324 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:57:40.147333 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:57:40.147351 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:57:40.147758 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:57:40.148101 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:57:40.148106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:57:40.148110 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:57:40.148285 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:57:40.148784 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:57:40.148792 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:57:40.149108 (+ 316us) client_negotiation.cc:769] Sending connection context
0929 20:57:40.149362 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 20:57:40.149891 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":327,"threads_started":1}
I20260929 20:57:40.152488 17958 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39347
I20260929 20:57:40.152809 17958 heartbeater.cc:464] Registering TS with master...
I20260929 20:57:40.153518 17958 heartbeater.cc:511] Master 127.0.97.62:39347 requested a full tablet report, sending...
I20260929 20:57:40.154898 17501 ts_manager.cc:194] Registered new tserver with Master: cfc4d0be29d54acfb46e90ff584cff75 (127.0.97.3:36431)
I20260929 20:57:40.155735 17501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52267
I20260929 20:57:40.162675 17691 heartbeater.cc:503] Master 127.0.97.62:39347 was elected leader, sending a full tablet report...
I20260929 20:57:40.165484 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:57:40.182056 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:40.169759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.0.97.62:39347)
0929 20:57:40.169954 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:40.169960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:40.170386 (+ 426us) server_negotiation.cc:407] Connection header received
0929 20:57:40.170517 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:40.170520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:40.170583 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:40.170661 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:40.171981 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.172710 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.174102 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.174583 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.174690 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:40.176952 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:40.176957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:40.176960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:40.176994 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:40.179611 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:40.180318 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:40.180322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:40.180324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:40.180448 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:40.180747 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:40.180750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:40.180752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:40.181096 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:40.181312 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:40.181569 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 20:57:40.181749 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:57:40.186286 17501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:57:40.193032 17501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:57:40.226076 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:40.205888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.0.97.2:36293)
0929 20:57:40.216961 (+ 11073us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:40.216971 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:40.218665 (+ 1694us) server_negotiation.cc:407] Connection header received
0929 20:57:40.218813 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:40.218819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:40.219017 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:40.219193 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:40.219899 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.221252 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.223955 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.224957 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.225072 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:40.225278 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:40.225599 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 20:57:40.225761 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10911,"thread_start_us":10774,"threads_started":1}
I20260929 20:57:40.226156 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:40.205252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36293 (local address 127.0.0.1:59582)
0929 20:57:40.218532 (+ 13280us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:40.218547 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:40.218663 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:40.219217 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:40.219224 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:40.219251 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:40.219693 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:40.219709 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.221455 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.221461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:40.223793 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:40.223806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.225153 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.225159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:40.225375 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:40.225414 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 20:57:40.225530 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 20:57:40.225725 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13144,"mutex_wait_us":55,"thread_start_us":12365,"threads_started":1}
I20260929 20:57:40.226833 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:40.205174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46773 (local address 127.0.0.1:44830)
0929 20:57:40.218238 (+ 13064us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:40.218297 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:40.218509 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:40.219535 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:40.219540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:40.219568 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:40.219972 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:40.219985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.221738 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.221747 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:40.223941 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:40.223954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.226044 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.226052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:40.226239 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:40.226272 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:57:40.226364 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 20:57:40.226491 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12745,"thread_start_us":13015,"threads_started":1}
I20260929 20:57:40.227188 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:40.205476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44830 (local address 127.0.97.1:46773)
0929 20:57:40.217360 (+ 11884us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:40.217370 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:40.218445 (+ 1075us) server_negotiation.cc:407] Connection header received
0929 20:57:40.218800 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:40.218810 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:40.219060 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:40.219245 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:40.220177 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.221616 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.224094 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.225549 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.225670 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:40.225907 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:40.226723 (+ 816us) server_negotiation.cc:299] Negotiation successful
0929 20:57:40.226905 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11729,"thread_start_us":11594,"threads_started":1}
I20260929 20:57:40.228446 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:40.218961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43690 (local address 127.0.97.3:36431)
0929 20:57:40.219859 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:40.219868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:40.220503 (+ 635us) server_negotiation.cc:407] Connection header received
0929 20:57:40.220693 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:40.220701 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:40.220939 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:40.221115 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:40.221856 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.223507 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.225851 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.227265 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.227371 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:40.227579 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:40.227964 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 20:57:40.228122 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":557,"threads_started":1}
I20260929 20:57:40.229270 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:40.219859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36431 (local address 127.0.0.1:43690)
0929 20:57:40.220381 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:40.220399 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:40.220513 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:40.221116 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:40.221120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:40.221143 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:40.221580 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:40.221591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.223659 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.223665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:40.225675 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:40.225688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.227677 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.227681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:40.227791 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:40.227822 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:57:40.227896 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 20:57:40.228016 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":272,"threads_started":1}
I20260929 20:57:40.230638 17760 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf5886491544533b925c1ddd732b0d0 (DEFAULT_TABLE table=test-table [id=06bd650c111b447182f7110eb230e06b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:57:40.230758 17626 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf5886491544533b925c1ddd732b0d0 (DEFAULT_TABLE table=test-table [id=06bd650c111b447182f7110eb230e06b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:57:40.232204 17626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf5886491544533b925c1ddd732b0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:40.232215 17760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf5886491544533b925c1ddd732b0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:40.232240 17893 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf5886491544533b925c1ddd732b0d0 (DEFAULT_TABLE table=test-table [id=06bd650c111b447182f7110eb230e06b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:57:40.233423 17893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf5886491544533b925c1ddd732b0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:40.245990 17976 tablet_bootstrap.cc:492] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75: Bootstrap starting.
I20260929 20:57:40.246507 17977 tablet_bootstrap.cc:492] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199: Bootstrap starting.
I20260929 20:57:40.248104 17978 tablet_bootstrap.cc:492] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e: Bootstrap starting.
I20260929 20:57:40.249671 17976 tablet_bootstrap.cc:654] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:40.250109 17977 tablet_bootstrap.cc:654] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:40.250828 17978 tablet_bootstrap.cc:654] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:40.251005 17976 log.cc:824] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:40.251287 17977 log.cc:824] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:40.251791 17978 log.cc:824] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:40.254328 17977 tablet_bootstrap.cc:492] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199: No bootstrap required, opened a new log
I20260929 20:57:40.254614 17977 ts_tablet_manager.cc:1397] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260929 20:57:40.254976 17978 tablet_bootstrap.cc:492] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e: No bootstrap required, opened a new log
I20260929 20:57:40.255009 17976 tablet_bootstrap.cc:492] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75: No bootstrap required, opened a new log
I20260929 20:57:40.255219 17978 ts_tablet_manager.cc:1397] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260929 20:57:40.255254 17976 ts_tablet_manager.cc:1397] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 20:57:40.262167 17978 raft_consensus.cc:348] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0738387bc0e84a808cbc507528df9199" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46773 } } peers { permanent_uuid: "6acd799d7a374bef938f93f08d82c52e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36293 } } peers { permanent_uuid: "cfc4d0be29d54acfb46e90ff584cff75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36431 } }
I20260929 20:57:40.262632 17978 raft_consensus.cc:374] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:40.262732 17978 raft_consensus.cc:729] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6acd799d7a374bef938f93f08d82c52e, State: Initialized, Role: FOLLOWER
I20260929 20:57:40.263288 17978 consensus_queue.cc:260] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0738387bc0e84a808cbc507528df9199" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46773 } } peers { permanent_uuid: "6acd799d7a374bef938f93f08d82c52e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36293 } } peers { permanent_uuid: "cfc4d0be29d54acfb46e90ff584cff75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36431 } }
I20260929 20:57:40.264331 17825 heartbeater.cc:503] Master 127.0.97.62:39347 was elected leader, sending a full tablet report...
I20260929 20:57:40.265039 17978 ts_tablet_manager.cc:1428] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e: Time spent starting tablet: real 0.010s user 0.003s sys 0.006s
I20260929 20:57:40.264873 17977 raft_consensus.cc:348] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0738387bc0e84a808cbc507528df9199" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46773 } } peers { permanent_uuid: "6acd799d7a374bef938f93f08d82c52e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36293 } } peers { permanent_uuid: "cfc4d0be29d54acfb46e90ff584cff75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36431 } }
I20260929 20:57:40.265267 17977 raft_consensus.cc:374] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:40.265367 17977 raft_consensus.cc:729] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0738387bc0e84a808cbc507528df9199, State: Initialized, Role: FOLLOWER
I20260929 20:57:40.265712 17976 raft_consensus.cc:348] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0738387bc0e84a808cbc507528df9199" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46773 } } peers { permanent_uuid: "6acd799d7a374bef938f93f08d82c52e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36293 } } peers { permanent_uuid: "cfc4d0be29d54acfb46e90ff584cff75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36431 } }
I20260929 20:57:40.266160 17976 raft_consensus.cc:374] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:40.266017 17977 consensus_queue.cc:260] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0738387bc0e84a808cbc507528df9199" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46773 } } peers { permanent_uuid: "6acd799d7a374bef938f93f08d82c52e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36293 } } peers { permanent_uuid: "cfc4d0be29d54acfb46e90ff584cff75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36431 } }
I20260929 20:57:40.266274 17976 raft_consensus.cc:729] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc4d0be29d54acfb46e90ff584cff75, State: Initialized, Role: FOLLOWER
I20260929 20:57:40.266934 17976 consensus_queue.cc:260] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0738387bc0e84a808cbc507528df9199" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46773 } } peers { permanent_uuid: "6acd799d7a374bef938f93f08d82c52e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36293 } } peers { permanent_uuid: "cfc4d0be29d54acfb46e90ff584cff75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36431 } }
I20260929 20:57:40.268108 17958 heartbeater.cc:503] Master 127.0.97.62:39347 was elected leader, sending a full tablet report...
I20260929 20:57:40.268895 17976 ts_tablet_manager.cc:1428] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 20:57:40.270444 17977 ts_tablet_manager.cc:1428] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
I20260929 20:57:40.271101 17983 raft_consensus.cc:482] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:57:40.271387 17983 raft_consensus.cc:504] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0738387bc0e84a808cbc507528df9199" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46773 } } peers { permanent_uuid: "6acd799d7a374bef938f93f08d82c52e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36293 } } peers { permanent_uuid: "cfc4d0be29d54acfb46e90ff584cff75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36431 } }
I20260929 20:57:40.274370 17983 leader_election.cc:288] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6acd799d7a374bef938f93f08d82c52e (127.0.97.2:36293), cfc4d0be29d54acfb46e90ff584cff75 (127.0.97.3:36431)
I20260929 20:57:40.280367 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:40.274015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38179 (local address 127.0.97.2:36293)
0929 20:57:40.274532 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:40.274539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:40.275062 (+ 523us) server_negotiation.cc:407] Connection header received
0929 20:57:40.275223 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:40.275228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:40.275291 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:40.275361 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:40.275898 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.276729 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.278601 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.279438 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.279554 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:40.279652 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:40.279982 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 20:57:40.280107 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357}
I20260929 20:57:40.280551 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:40.273477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36293 (local address 127.0.97.1:38179)
0929 20:57:40.274966 (+ 1489us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:40.274981 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:40.275103 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:40.275389 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:40.275394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:40.275418 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:40.275733 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:40.275743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.276846 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.276851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:40.278463 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:40.278476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.279658 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.279665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:40.279837 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:40.279861 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:57:40.279940 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 20:57:40.280123 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1367,"mutex_wait_us":20,"thread_start_us":2595,"threads_started":1}
I20260929 20:57:40.281754 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:40.273630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34449 (local address 127.0.97.3:36431)
0929 20:57:40.273862 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:40.273868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:40.276469 (+ 2601us) server_negotiation.cc:407] Connection header received
0929 20:57:40.276613 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:40.276618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:40.276673 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:40.276742 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:40.277364 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.278101 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.279819 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.280857 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.280971 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:40.281066 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:40.281382 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 20:57:40.281519 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260929 20:57:40.281867 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:40.273502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36431 (local address 127.0.97.1:34449)
0929 20:57:40.276369 (+ 2867us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:40.276387 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:40.276503 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:40.276769 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:40.276773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:40.276796 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:40.277214 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:40.277224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.278258 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.278264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:40.279664 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:40.279676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.281053 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.281059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:40.281233 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:40.281265 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:57:40.281339 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 20:57:40.281492 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2750,"mutex_wait_us":2,"thread_start_us":1152,"threads_started":1}
I20260929 20:57:40.281930 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf5886491544533b925c1ddd732b0d0" candidate_uuid: "0738387bc0e84a808cbc507528df9199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6acd799d7a374bef938f93f08d82c52e" is_pre_election: true
I20260929 20:57:40.282477 17780 raft_consensus.cc:2454] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0738387bc0e84a808cbc507528df9199 in term 0.
I20260929 20:57:40.282799 17913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf5886491544533b925c1ddd732b0d0" candidate_uuid: "0738387bc0e84a808cbc507528df9199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc4d0be29d54acfb46e90ff584cff75" is_pre_election: true
I20260929 20:57:40.283126 17582 leader_election.cc:302] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0738387bc0e84a808cbc507528df9199, 6acd799d7a374bef938f93f08d82c52e; no voters:
I20260929 20:57:40.283319 17913 raft_consensus.cc:2454] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0738387bc0e84a808cbc507528df9199 in term 0.
I20260929 20:57:40.283481 17983 raft_consensus.cc:2781] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:57:40.283615 17983 raft_consensus.cc:482] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:57:40.283725 17983 raft_consensus.cc:3037] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:40.286075 17983 raft_consensus.cc:504] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0738387bc0e84a808cbc507528df9199" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46773 } } peers { permanent_uuid: "6acd799d7a374bef938f93f08d82c52e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36293 } } peers { permanent_uuid: "cfc4d0be29d54acfb46e90ff584cff75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36431 } }
I20260929 20:57:40.286744 17983 leader_election.cc:288] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [CANDIDATE]: Term 1 election: Requested vote from peers 6acd799d7a374bef938f93f08d82c52e (127.0.97.2:36293), cfc4d0be29d54acfb46e90ff584cff75 (127.0.97.3:36431)
I20260929 20:57:40.287176 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf5886491544533b925c1ddd732b0d0" candidate_uuid: "0738387bc0e84a808cbc507528df9199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6acd799d7a374bef938f93f08d82c52e"
I20260929 20:57:40.287391 17780 raft_consensus.cc:3037] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:40.287673 17913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf5886491544533b925c1ddd732b0d0" candidate_uuid: "0738387bc0e84a808cbc507528df9199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc4d0be29d54acfb46e90ff584cff75"
I20260929 20:57:40.287882 17913 raft_consensus.cc:3037] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:40.290706 17780 raft_consensus.cc:2454] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0738387bc0e84a808cbc507528df9199 in term 1.
I20260929 20:57:40.290987 17913 raft_consensus.cc:2454] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0738387bc0e84a808cbc507528df9199 in term 1.
I20260929 20:57:40.291163 17582 leader_election.cc:302] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0738387bc0e84a808cbc507528df9199, 6acd799d7a374bef938f93f08d82c52e; no voters:
I20260929 20:57:40.291476 17983 raft_consensus.cc:2781] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:57:40.291906 17983 raft_consensus.cc:686] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [term 1 LEADER]: Becoming Leader. State: Replica: 0738387bc0e84a808cbc507528df9199, State: Running, Role: LEADER
I20260929 20:57:40.292426 17983 consensus_queue.cc:237] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0738387bc0e84a808cbc507528df9199" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46773 } } peers { permanent_uuid: "6acd799d7a374bef938f93f08d82c52e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36293 } } peers { permanent_uuid: "cfc4d0be29d54acfb46e90ff584cff75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36431 } }
I20260929 20:57:40.296777 17501 catalog_manager.cc:5495] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0738387bc0e84a808cbc507528df9199 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "0738387bc0e84a808cbc507528df9199" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0738387bc0e84a808cbc507528df9199" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6acd799d7a374bef938f93f08d82c52e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfc4d0be29d54acfb46e90ff584cff75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36431 } health_report { overall_health: UNKNOWN } } }
I20260929 20:57:40.318781 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:40.313719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44840 (local address 127.0.97.1:46773)
0929 20:57:40.313931 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:40.313936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:40.313954 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:57:40.314124 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:40.314127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:40.314264 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:40.314389 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:57:40.314694 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.315445 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.316777 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.317178 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.317288 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:40.317675 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:57:40.318263 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:57:40.318373 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:40.318463 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 20:57:40.318531 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:57:40.327157 17913 raft_consensus.cc:1261] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75 [term 1 FOLLOWER]: Refusing update from remote peer 0738387bc0e84a808cbc507528df9199: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:57:40.327157 17780 raft_consensus.cc:1261] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e [term 1 FOLLOWER]: Refusing update from remote peer 0738387bc0e84a808cbc507528df9199: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:57:40.328047 17987 consensus_queue.cc:1035] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfc4d0be29d54acfb46e90ff584cff75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:57:40.328439 17983 consensus_queue.cc:1035] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6acd799d7a374bef938f93f08d82c52e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:57:40.339389 17990 mvcc.cc:205] Tried to move back new op lower bound from 7334770525489426432 to 7334770525361602560. Current Snapshot: MvccSnapshot[applied={T|T < 7334770525489426432}]
I20260929 20:57:40.353408 17500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:57:40.353751 17500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:57:40.357712 17500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c23d88eee5314f33bb79aca5e2f7bd99: Sending DeleteTablet for 3 replicas of tablet 1cf5886491544533b925c1ddd732b0d0
I20260929 20:57:40.358829 17626 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf5886491544533b925c1ddd732b0d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:57:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44830
I20260929 20:57:40.358858 17760 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf5886491544533b925c1ddd732b0d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:57:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59582
I20260929 20:57:40.359851 18000 tablet_replica.cc:330] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e: stopping tablet replica
I20260929 20:57:40.359779 17893 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf5886491544533b925c1ddd732b0d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:57:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43690
I20260929 20:57:40.360320 18000 raft_consensus.cc:2229] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:57:40.360841 18000 raft_consensus.cc:2258] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:57:40.361009 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 17563
I20260929 20:57:40.365303 18000 ts_tablet_manager.cc:1905] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:57:40.373317 18000 ts_tablet_manager.cc:1918] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:57:40.373698 18000 log.cc:1196] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/1cf5886491544533b925c1ddd732b0d0
I20260929 20:57:40.374414 18000 ts_tablet_manager.cc:1939] T 1cf5886491544533b925c1ddd732b0d0 P 6acd799d7a374bef938f93f08d82c52e: Deleting consensus metadata
I20260929 20:57:40.376427 17489 catalog_manager.cc:4841] TS 6acd799d7a374bef938f93f08d82c52e (127.0.97.2:36293): tablet 1cf5886491544533b925c1ddd732b0d0 (table test-table [id=06bd650c111b447182f7110eb230e06b]) successfully deleted
I20260929 20:57:40.377439 18002 tablet_replica.cc:330] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75: stopping tablet replica
I20260929 20:57:40.377943 18002 raft_consensus.cc:2229] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:57:40.378564 18002 raft_consensus.cc:2258] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:57:40.379185 18001 tablet_replica.cc:330] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199: stopping tablet replica
I20260929 20:57:40.379631 18001 raft_consensus.cc:2229] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:57:40.381393 18001 raft_consensus.cc:2258] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:57:40.382390 18002 ts_tablet_manager.cc:1905] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:57:40.383522 18001 ts_tablet_manager.cc:1905] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 20:57:40 dist-test-slave-m9g1 krb5kdc[17457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:57:40.389570 18002 ts_tablet_manager.cc:1918] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:57:40.389843 18002 log.cc:1196] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/1cf5886491544533b925c1ddd732b0d0
I20260929 20:57:40.390446 18002 ts_tablet_manager.cc:1939] T 1cf5886491544533b925c1ddd732b0d0 P cfc4d0be29d54acfb46e90ff584cff75: Deleting consensus metadata
I20260929 20:57:40.391825 18001 ts_tablet_manager.cc:1918] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:57:40.391986 18001 log.cc:1196] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/1cf5886491544533b925c1ddd732b0d0
I20260929 20:57:40.392505 18001 ts_tablet_manager.cc:1939] T 1cf5886491544533b925c1ddd732b0d0 P 0738387bc0e84a808cbc507528df9199: Deleting consensus metadata
I20260929 20:57:40.392526 17486 catalog_manager.cc:4841] TS cfc4d0be29d54acfb46e90ff584cff75 (127.0.97.3:36431): tablet 1cf5886491544533b925c1ddd732b0d0 (table test-table [id=06bd650c111b447182f7110eb230e06b]) successfully deleted
I20260929 20:57:40.393411 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:40.362299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44850 (local address 127.0.97.1:46773)
0929 20:57:40.362477 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:40.362484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:40.373237 (+ 10753us) server_negotiation.cc:407] Connection header received
0929 20:57:40.373778 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:40.373794 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:40.373882 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:40.373978 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:40.375811 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.376976 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.378743 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.379516 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.379618 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:40.385560 (+ 5942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:40.385581 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:40.385599 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:40.385652 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:40.389757 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:40.390725 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:40.390735 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:40.390739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:40.390903 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:40.391396 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:40.391401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:40.391404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:40.391902 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:40.392207 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:40.392813 (+ 606us) server_negotiation.cc:299] Negotiation successful
0929 20:57:40.393052 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 20:57:40.394263 17486 catalog_manager.cc:4841] TS 0738387bc0e84a808cbc507528df9199 (127.0.97.1:46773): tablet 1cf5886491544533b925c1ddd732b0d0 (table test-table [id=06bd650c111b447182f7110eb230e06b]) successfully deleted
I20260929 20:57:40.394807 17686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:40.493989 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 17563
I20260929 20:57:40.509393 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 17697
Sep 29 20:57:40 dist-test-slave-m9g1 krb5kdc[17457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:57:40.524794 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:40.510308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.0.97.2:36293)
0929 20:57:40.510494 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:40.510500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:40.510523 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:57:40.510587 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:40.510590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:40.510653 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:40.510747 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:40.511961 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.512710 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.514116 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.514567 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.514682 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:40.519144 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:40.519161 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:40.519177 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:40.519215 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:40.521866 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:40.522705 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:40.522713 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:40.522715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:40.522850 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:40.523209 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:40.523212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:40.523215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:40.523614 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:40.523900 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:40.524229 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 20:57:40.524482 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:57:40.525470 17820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:40.615985 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 17697
I20260929 20:57:40.629663 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 17830
Sep 29 20:57:40 dist-test-slave-m9g1 krb5kdc[17457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:57:40.644170 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:40.630421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43694 (local address 127.0.97.3:36431)
0929 20:57:40.630595 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:40.630601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:40.630620 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:57:40.630842 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:40.630846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:40.630914 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:40.630999 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:40.632245 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.632968 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.634345 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:40.634737 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:40.634859 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:40.638987 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:40.639004 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:40.639019 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:40.639057 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:40.641555 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:40.642153 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:40.642160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:40.642162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:40.642403 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:40.642734 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:40.642739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:40.642741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:40.643091 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:40.643344 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:40.643652 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 20:57:40.643870 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:57:40.644757 17953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:40.732308 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 17830
I20260929 20:57:40.745599 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 17470
I20260929 20:57:40.746789 17532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:40.856781 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 17470
2026-09-29T20:57:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4012 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T20:57:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:57:40Z Disabled control of system clock
I20260929 20:57:40.925583 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39737
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:57:41.247089 18011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:41.247392 18011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:41.247493 18011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:41.254801 18011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:57:41.254920 18011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:41.254969 18011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:41.255028 18011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:57:41.255070 18011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:57:41.264364 18011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39737
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:41.265620 18011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:41.267436 18011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:41.278566 18016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:41.278586 18017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:41.278689 18019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:41.279246 18011 server_base.cc:1034] running on GCE node
I20260929 20:57:41.280023 18011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:41.281424 18011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:41.282629 18011 hybrid_clock.cc:648] HybridClock initialized: now 1790715461282572 us; error 49 us; skew 500 ppm
I20260929 20:57:41.282970 18011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:57:41.285151 18011 webserver.cc:460] Webserver started at http://127.0.97.62:41303/ using document root <none> and password file <none>
I20260929 20:57:41.285825 18011 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:41.285923 18011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:41.286262 18011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:41.289165 18011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "1241cfc8d30f4bb7890d3c9c86449184"
format_stamp: "Formatted at 2026-09-29 20:57:41 on dist-test-slave-m9g1"
server_key: "21510191f3ad13de4116bb7bc2eb1503"
server_key_iv: "7e80bb9686ef6f982767d3a75a02b163"
server_key_version: "encryptionkey@0"
I20260929 20:57:41.289932 18011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "1241cfc8d30f4bb7890d3c9c86449184"
format_stamp: "Formatted at 2026-09-29 20:57:41 on dist-test-slave-m9g1"
server_key: "21510191f3ad13de4116bb7bc2eb1503"
server_key_iv: "7e80bb9686ef6f982767d3a75a02b163"
server_key_version: "encryptionkey@0"
I20260929 20:57:41.295056 18011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 20:57:41.298424 18025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:41.299695 18011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:57:41.299880 18011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "1241cfc8d30f4bb7890d3c9c86449184"
format_stamp: "Formatted at 2026-09-29 20:57:41 on dist-test-slave-m9g1"
server_key: "21510191f3ad13de4116bb7bc2eb1503"
server_key_iv: "7e80bb9686ef6f982767d3a75a02b163"
server_key_version: "encryptionkey@0"
I20260929 20:57:41.300066 18011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:41.316553 18011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:41.317365 18011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:41.317613 18011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:41.334494 18011 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39737
I20260929 20:57:41.334501 18076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39737 every 8 connection(s)
I20260929 20:57:41.335822 18011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:57:41.337620 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 18011
I20260929 20:57:41.340405 18077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:41.342072 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:57:41.342674 388 external_mini_cluster.cc:1428] Setting key 0b7b2bbbd98739f46b3c9151e8c13f29
I20260929 20:57:41.353613 18077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184: Bootstrap starting.
I20260929 20:57:41.356011 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:41.344637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57696 (local address 127.0.97.62:39737)
0929 20:57:41.345696 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:41.345715 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:41.345759 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:57:41.347446 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:41.347470 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:41.347985 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:41.348377 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:41.348922 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:41.350211 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:41.351636 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:41.352362 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:41.352533 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:41.353700 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:41.353710 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:41.353729 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:41.354342 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:41.354426 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:41.354448 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:41.354553 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:41.354764 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 20:57:41.355082 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":372,"threads_started":1}
I20260929 20:57:41.356894 18077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:41.357897 18077 log.cc:824] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:41.360602 18077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184: No bootstrap required, opened a new log
I20260929 20:57:41.367769 18077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1241cfc8d30f4bb7890d3c9c86449184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } }
I20260929 20:57:41.368120 18077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:41.368211 18077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1241cfc8d30f4bb7890d3c9c86449184, State: Initialized, Role: FOLLOWER
I20260929 20:57:41.368753 18077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1241cfc8d30f4bb7890d3c9c86449184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } }
I20260929 20:57:41.368968 18077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:57:41.369086 18077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:57:41.369235 18077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:41.371511 18077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1241cfc8d30f4bb7890d3c9c86449184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } }
I20260929 20:57:41.372001 18077 leader_election.cc:302] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1241cfc8d30f4bb7890d3c9c86449184; no voters:
I20260929 20:57:41.372545 18077 leader_election.cc:288] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:57:41.372677 18082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:57:41.373380 18082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184 [term 1 LEADER]: Becoming Leader. State: Replica: 1241cfc8d30f4bb7890d3c9c86449184, State: Running, Role: LEADER
I20260929 20:57:41.373914 18082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1241cfc8d30f4bb7890d3c9c86449184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } }
I20260929 20:57:41.374511 18077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:57:41.376454 18083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1241cfc8d30f4bb7890d3c9c86449184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1241cfc8d30f4bb7890d3c9c86449184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } }
I20260929 20:57:41.376876 18083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184 [sys.catalog]: This master's current role is: LEADER
I20260929 20:57:41.376461 18084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1241cfc8d30f4bb7890d3c9c86449184. Latest consensus state: current_term: 1 leader_uuid: "1241cfc8d30f4bb7890d3c9c86449184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1241cfc8d30f4bb7890d3c9c86449184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } }
I20260929 20:57:41.377521 18084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184 [sys.catalog]: This master's current role is: LEADER
I20260929 20:57:41.378743 18089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:57:41.384228 18089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:57:41.392998 18089 catalog_manager.cc:1347] Generated new cluster ID: f430968d228c413c9236df9edab9a676
I20260929 20:57:41.393102 18089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:57:41.417693 18089 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:57:41.419142 18089 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:57:41.429711 18089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184: Generated new TSK 0
I20260929 20:57:41.430594 18089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:57:41.441653 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39737
--builtin_ntp_servers=127.0.97.20:45405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:57:41.767040 18101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:41.767297 18101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:41.767390 18101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:41.774574 18101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:41.774667 18101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:41.774775 18101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:57:41.784017 18101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:41.785254 18101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:41.787075 18101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:41.799523 18107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:41.799456 18106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:41.799909 18109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:41.800254 18101 server_base.cc:1034] running on GCE node
I20260929 20:57:41.800803 18101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:41.801713 18101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:41.802937 18101 hybrid_clock.cc:648] HybridClock initialized: now 1790715461802881 us; error 41 us; skew 500 ppm
I20260929 20:57:41.803234 18101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:57:41.810566 18101 webserver.cc:460] Webserver started at http://127.0.97.1:38171/ using document root <none> and password file <none>
I20260929 20:57:41.811252 18101 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:41.811358 18101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:41.811659 18101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:41.814513 18101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "d2f55ec0c2164972938745491569301e"
format_stamp: "Formatted at 2026-09-29 20:57:41 on dist-test-slave-m9g1"
server_key: "cc6f040dc15d0666a8d11bbe2647a975"
server_key_iv: "4dca9d5be8e959c786ce2857f9612fca"
server_key_version: "encryptionkey@0"
I20260929 20:57:41.815254 18101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d2f55ec0c2164972938745491569301e"
format_stamp: "Formatted at 2026-09-29 20:57:41 on dist-test-slave-m9g1"
server_key: "cc6f040dc15d0666a8d11bbe2647a975"
server_key_iv: "4dca9d5be8e959c786ce2857f9612fca"
server_key_version: "encryptionkey@0"
I20260929 20:57:41.820252 18101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 20:57:41.823580 18115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:41.824903 18101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:57:41.825042 18101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "d2f55ec0c2164972938745491569301e"
format_stamp: "Formatted at 2026-09-29 20:57:41 on dist-test-slave-m9g1"
server_key: "cc6f040dc15d0666a8d11bbe2647a975"
server_key_iv: "4dca9d5be8e959c786ce2857f9612fca"
server_key_version: "encryptionkey@0"
I20260929 20:57:41.825229 18101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:41.856221 18101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:41.857062 18101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:41.857312 18101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:41.858269 18101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:41.859750 18101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:41.859843 18101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:41.859954 18101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:41.860023 18101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:41.889454 18101 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36585
I20260929 20:57:41.889509 18227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36585 every 8 connection(s)
I20260929 20:57:41.890766 18101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:57:41.894065 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 18101
I20260929 20:57:41.894263 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:57:41.894812 388 external_mini_cluster.cc:1428] Setting key e6452e27eb772c4c82fb31940c6d835f
I20260929 20:57:41.897513 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39737
--builtin_ntp_servers=127.0.97.20:45405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:57:41.903438 18230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:41.893289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53739 (local address 127.0.97.62:39737)
0929 20:57:41.893754 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:41.893759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:41.895069 (+ 1310us) server_negotiation.cc:407] Connection header received
0929 20:57:41.897034 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:41.897039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:41.897121 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:41.897224 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:41.898404 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:41.899145 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:41.900662 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:41.901062 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:41.901173 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:41.902580 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:41.902585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:41.902588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:41.902607 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:41.902648 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:41.902663 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:41.902788 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:41.903069 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 20:57:41.903157 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":212,"threads_started":1}
I20260929 20:57:41.904528 18231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:41.893927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.1:53739)
0929 20:57:41.894919 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:41.894955 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:41.896766 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:41.897470 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:41.897479 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:41.897574 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:57:41.898240 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:41.898261 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:41.899289 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:41.899293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:41.900520 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:41.900532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:41.901289 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:41.901297 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:41.901526 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:41.902259 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:57:41.902261 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:57:41.902308 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:57:41.902392 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:57:41.902809 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:57:41.902820 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:57:41.902829 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 20:57:41.903031 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 20:57:41.903447 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":328,"threads_started":1}
I20260929 20:57:41.906062 18228 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39737
I20260929 20:57:41.906378 18228 heartbeater.cc:464] Registering TS with master...
I20260929 20:57:41.907112 18228 heartbeater.cc:511] Master 127.0.97.62:39737 requested a full tablet report, sending...
I20260929 20:57:41.909089 18042 ts_manager.cc:194] Registered new tserver with Master: d2f55ec0c2164972938745491569301e (127.0.97.1:36585)
I20260929 20:57:41.910605 18042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:53739
W20260929 20:57:42.216841 18232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:42.217124 18232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:42.217242 18232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:42.224603 18232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:42.224731 18232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:42.224916 18232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:57:42.234159 18232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:42.235405 18232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:42.237138 18232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:42.249758 18238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:42.249782 18237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:42.250347 18240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:42.250659 18232 server_base.cc:1034] running on GCE node
I20260929 20:57:42.251284 18232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:42.252279 18232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:42.253494 18232 hybrid_clock.cc:648] HybridClock initialized: now 1790715462253431 us; error 59 us; skew 500 ppm
I20260929 20:57:42.253823 18232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:57:42.256039 18232 webserver.cc:460] Webserver started at http://127.0.97.2:46451/ using document root <none> and password file <none>
I20260929 20:57:42.256709 18232 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:42.256836 18232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:42.257143 18232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:42.260046 18232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "93131799a2b0463a9d2bca2c9f0d5f7a"
format_stamp: "Formatted at 2026-09-29 20:57:42 on dist-test-slave-m9g1"
server_key: "188b981a587fa6496a27daae3d4bf689"
server_key_iv: "779796d76e9548b9efc00e341a0c6abd"
server_key_version: "encryptionkey@0"
I20260929 20:57:42.260789 18232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "93131799a2b0463a9d2bca2c9f0d5f7a"
format_stamp: "Formatted at 2026-09-29 20:57:42 on dist-test-slave-m9g1"
server_key: "188b981a587fa6496a27daae3d4bf689"
server_key_iv: "779796d76e9548b9efc00e341a0c6abd"
server_key_version: "encryptionkey@0"
I20260929 20:57:42.265892 18232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:57:42.269285 18246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:42.270690 18232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:57:42.270848 18232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "93131799a2b0463a9d2bca2c9f0d5f7a"
format_stamp: "Formatted at 2026-09-29 20:57:42 on dist-test-slave-m9g1"
server_key: "188b981a587fa6496a27daae3d4bf689"
server_key_iv: "779796d76e9548b9efc00e341a0c6abd"
server_key_version: "encryptionkey@0"
I20260929 20:57:42.271044 18232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:42.295053 18232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:42.295897 18232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:42.296157 18232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:42.297112 18232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:42.298610 18232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:42.298702 18232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:42.298804 18232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:42.298874 18232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:42.328259 18232 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42785
I20260929 20:57:42.328290 18358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42785 every 8 connection(s)
I20260929 20:57:42.329624 18232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:57:42.330396 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 18232
I20260929 20:57:42.330569 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:57:42.331084 388 external_mini_cluster.cc:1428] Setting key 32a1b23072558c63400df0841761dca3
I20260929 20:57:42.334030 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39737
--builtin_ntp_servers=127.0.97.20:45405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:57:42.344542 18230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:42.332893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41051 (local address 127.0.97.62:39737)
0929 20:57:42.333087 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:42.333093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:42.334299 (+ 1206us) server_negotiation.cc:407] Connection header received
0929 20:57:42.336683 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:42.336689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:42.336768 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:42.336860 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:42.338260 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.339394 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.341447 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.342078 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.342210 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:42.343756 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:42.343761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:42.343764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:42.343782 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:42.343813 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:42.343825 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:42.343902 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:42.344122 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 20:57:42.344230 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:57:42.345660 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:42.332988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.2:41051)
0929 20:57:42.334116 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:42.334175 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:42.336377 (+ 2202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:42.337174 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:42.337185 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:42.337317 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:57:42.338065 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:42.338088 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.339543 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.339549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:42.341293 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:42.341309 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.342351 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.342358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:42.342638 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:42.343420 (+ 782us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:57:42.343423 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:57:42.343466 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:57:42.343561 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:57:42.343914 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:57:42.343925 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:57:42.343933 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:57:42.344090 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 20:57:42.344502 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":301,"threads_started":1}
I20260929 20:57:42.347093 18359 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39737
I20260929 20:57:42.347405 18359 heartbeater.cc:464] Registering TS with master...
I20260929 20:57:42.348160 18359 heartbeater.cc:511] Master 127.0.97.62:39737 requested a full tablet report, sending...
I20260929 20:57:42.349592 18042 ts_manager.cc:194] Registered new tserver with Master: 93131799a2b0463a9d2bca2c9f0d5f7a (127.0.97.2:42785)
I20260929 20:57:42.350461 18042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:41051
W20260929 20:57:42.657398 18362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:42.657698 18362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:42.657796 18362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:42.665409 18362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:42.665537 18362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:42.665648 18362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:57:42.675504 18362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:42.676832 18362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:42.678696 18362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:42.692075 18368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:42.692157 18370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:42.692128 18367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:42.692956 18362 server_base.cc:1034] running on GCE node
I20260929 20:57:42.693578 18362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:42.694639 18362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:42.695881 18362 hybrid_clock.cc:648] HybridClock initialized: now 1790715462695820 us; error 57 us; skew 500 ppm
I20260929 20:57:42.696228 18362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:57:42.698596 18362 webserver.cc:460] Webserver started at http://127.0.97.3:42729/ using document root <none> and password file <none>
I20260929 20:57:42.699342 18362 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:42.699453 18362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:42.699750 18362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:42.702904 18362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "c8e680426e92482c90ef7e462568e123"
format_stamp: "Formatted at 2026-09-29 20:57:42 on dist-test-slave-m9g1"
server_key: "f451fef82a08d798dc6fd3bdab5deba5"
server_key_iv: "b771466e261a73b00eaddee106c97fb8"
server_key_version: "encryptionkey@0"
I20260929 20:57:42.703804 18362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c8e680426e92482c90ef7e462568e123"
format_stamp: "Formatted at 2026-09-29 20:57:42 on dist-test-slave-m9g1"
server_key: "f451fef82a08d798dc6fd3bdab5deba5"
server_key_iv: "b771466e261a73b00eaddee106c97fb8"
server_key_version: "encryptionkey@0"
I20260929 20:57:42.709102 18362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:57:42.712702 18376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:42.714097 18362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:57:42.714287 18362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "c8e680426e92482c90ef7e462568e123"
format_stamp: "Formatted at 2026-09-29 20:57:42 on dist-test-slave-m9g1"
server_key: "f451fef82a08d798dc6fd3bdab5deba5"
server_key_iv: "b771466e261a73b00eaddee106c97fb8"
server_key_version: "encryptionkey@0"
I20260929 20:57:42.714514 18362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:42.731455 18362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:42.732293 18362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:42.732533 18362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:42.733479 18362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:42.734938 18362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:42.735024 18362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:42.735096 18362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:42.735148 18362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:42.763827 18362 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34065
I20260929 20:57:42.763880 18488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34065 every 8 connection(s)
I20260929 20:57:42.765074 18362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:57:42.771622 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 18362
I20260929 20:57:42.771795 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:57:42.772382 388 external_mini_cluster.cc:1428] Setting key de7bd4d20022fdb2f645f9978177c18f
I20260929 20:57:42.778512 18230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:42.767932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34793 (local address 127.0.97.62:39737)
0929 20:57:42.768191 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:42.768197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:42.770016 (+ 1819us) server_negotiation.cc:407] Connection header received
0929 20:57:42.771861 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:42.771867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:42.771955 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:42.772063 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:42.773226 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.774142 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.776057 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.776475 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.776572 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:42.777750 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:42.777754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:42.777758 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:42.777777 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:42.777810 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:42.777822 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:42.777891 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:42.778123 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 20:57:42.778241 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:57:42.779486 18491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:42.768641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.3:34793)
0929 20:57:42.769856 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:42.769895 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:42.771529 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:42.772264 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:42.772273 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:42.772367 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:57:42.773050 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:42.773076 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.774381 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.774386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:42.775938 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:42.775950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.776647 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.776652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:42.776851 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:42.777459 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:57:42.777462 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:57:42.777502 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:57:42.777576 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:57:42.777932 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:57:42.777943 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:57:42.777950 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:57:42.778105 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 20:57:42.778470 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":427,"threads_started":1}
I20260929 20:57:42.780740 18489 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39737
I20260929 20:57:42.781023 18489 heartbeater.cc:464] Registering TS with master...
I20260929 20:57:42.781749 18489 heartbeater.cc:511] Master 127.0.97.62:39737 requested a full tablet report, sending...
I20260929 20:57:42.783078 18042 ts_manager.cc:194] Registered new tserver with Master: c8e680426e92482c90ef7e462568e123 (127.0.97.3:34065)
I20260929 20:57:42.783874 18042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:34793
I20260929 20:57:42.787746 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:57:42.797020 18230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:42.791667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.0.97.62:39737)
0929 20:57:42.791850 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:42.791854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:42.792136 (+ 282us) server_negotiation.cc:407] Connection header received
0929 20:57:42.792266 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:42.792269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:42.792326 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:42.792404 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:42.792853 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.793567 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.794879 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.795285 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.795380 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:42.796497 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:42.796502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:42.796505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:42.796523 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:42.796555 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:42.796567 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:42.796634 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:42.796716 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 20:57:42.796811 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:57:42.800665 18042 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:57:42.807523 18042 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:57:42.841797 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:42.819302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34065 (local address 127.0.0.1:55386)
0929 20:57:42.832132 (+ 12830us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:42.832180 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:42.832412 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:42.833761 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:42.833769 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:42.833803 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:42.834219 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:42.834233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.836376 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.836383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:42.839115 (+ 2732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:42.839132 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.840857 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.840862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:42.841093 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:42.841151 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 20:57:42.841278 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 20:57:42.841472 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12497,"thread_start_us":12274,"threads_started":1}
I20260929 20:57:42.841843 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:42.819848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55386 (local address 127.0.97.3:34065)
0929 20:57:42.832518 (+ 12670us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:42.832535 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:42.832572 (+ 37us) server_negotiation.cc:407] Connection header received
0929 20:57:42.832683 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:42.832692 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:42.832982 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:42.833438 (+ 456us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:42.834471 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.836193 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.839368 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.840666 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.840797 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:42.841082 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:42.841356 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 20:57:42.841522 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12496,"thread_start_us":13284,"threads_started":1}
I20260929 20:57:42.845170 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:42.819839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40288 (local address 127.0.97.2:42785)
0929 20:57:42.835396 (+ 15557us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:42.835407 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:42.837076 (+ 1669us) server_negotiation.cc:407] Connection header received
0929 20:57:42.837193 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:42.837202 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:42.837484 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:42.838465 (+ 981us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:42.838775 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.840397 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.842803 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.843864 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.843951 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:42.844139 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:42.844626 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 20:57:42.844788 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15357,"thread_start_us":15570,"threads_started":1}
I20260929 20:57:42.845381 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:42.826036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42785 (local address 127.0.0.1:40288)
0929 20:57:42.836811 (+ 10775us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:42.836832 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:42.836943 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:42.837694 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:42.837699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:42.837727 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:42.838213 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:42.838225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.840548 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.840553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:42.842587 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:42.842601 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.844300 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.844305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:42.844451 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:42.844474 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:57:42.844543 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 20:57:42.844662 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10649,"thread_start_us":10468,"threads_started":1}
I20260929 20:57:42.846349 18424 tablet_service.cc:1467] Processing CreateTablet for tablet 91e1c2ab2c874e60ad3d3e2d0624e806 (DEFAULT_TABLE table=test-table [id=4323cefcff1d4abdab2391bdcc36e0fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:57:42.847734 18424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e1c2ab2c874e60ad3d3e2d0624e806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:42.848783 18294 tablet_service.cc:1467] Processing CreateTablet for tablet 91e1c2ab2c874e60ad3d3e2d0624e806 (DEFAULT_TABLE table=test-table [id=4323cefcff1d4abdab2391bdcc36e0fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:57:42.849776 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:42.836874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36585 (local address 127.0.0.1:43172)
0929 20:57:42.840860 (+ 3986us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:42.840882 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:42.841005 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:42.842652 (+ 1647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:42.842656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:42.842679 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:42.842995 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:42.843003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.844786 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.844791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:42.847078 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:42.847094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.849097 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.849103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:42.849278 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:42.849309 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:57:42.849393 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 20:57:42.849509 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3850,"thread_start_us":4204,"threads_started":1}
I20260929 20:57:42.849858 18294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e1c2ab2c874e60ad3d3e2d0624e806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:42.850564 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:42.833023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43172 (local address 127.0.97.1:36585)
0929 20:57:42.840203 (+ 7180us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:42.840211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:42.842087 (+ 1876us) server_negotiation.cc:407] Connection header received
0929 20:57:42.842215 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:42.842223 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:42.842476 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:42.842647 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:42.843149 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.844626 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.847270 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:42.848625 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:42.848751 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:42.848984 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:42.850033 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0929 20:57:42.850226 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7022,"thread_start_us":6908,"threads_started":1}
I20260929 20:57:42.854125 18163 tablet_service.cc:1467] Processing CreateTablet for tablet 91e1c2ab2c874e60ad3d3e2d0624e806 (DEFAULT_TABLE table=test-table [id=4323cefcff1d4abdab2391bdcc36e0fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:57:42.855494 18163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e1c2ab2c874e60ad3d3e2d0624e806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:42.859529 18507 tablet_bootstrap.cc:492] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123: Bootstrap starting.
I20260929 20:57:42.863008 18507 tablet_bootstrap.cc:654] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:42.863432 18508 tablet_bootstrap.cc:492] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a: Bootstrap starting.
I20260929 20:57:42.864382 18507 log.cc:824] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:42.866577 18508 tablet_bootstrap.cc:654] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:42.867429 18507 tablet_bootstrap.cc:492] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123: No bootstrap required, opened a new log
I20260929 20:57:42.867686 18507 ts_tablet_manager.cc:1397] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 20:57:42.868284 18508 log.cc:824] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:42.870316 18510 tablet_bootstrap.cc:492] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e: Bootstrap starting.
I20260929 20:57:42.871209 18508 tablet_bootstrap.cc:492] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a: No bootstrap required, opened a new log
I20260929 20:57:42.871454 18508 ts_tablet_manager.cc:1397] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 20:57:42.872886 18510 tablet_bootstrap.cc:654] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:42.874326 18510 log.cc:824] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:42.877904 18507 raft_consensus.cc:348] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e680426e92482c90ef7e462568e123" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } } peers { permanent_uuid: "93131799a2b0463a9d2bca2c9f0d5f7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42785 } } peers { permanent_uuid: "d2f55ec0c2164972938745491569301e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36585 } }
I20260929 20:57:42.878383 18507 raft_consensus.cc:374] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:42.878476 18507 raft_consensus.cc:729] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8e680426e92482c90ef7e462568e123, State: Initialized, Role: FOLLOWER
I20260929 20:57:42.879215 18507 consensus_queue.cc:260] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e680426e92482c90ef7e462568e123" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } } peers { permanent_uuid: "93131799a2b0463a9d2bca2c9f0d5f7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42785 } } peers { permanent_uuid: "d2f55ec0c2164972938745491569301e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36585 } }
I20260929 20:57:42.884316 18510 tablet_bootstrap.cc:492] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e: No bootstrap required, opened a new log
I20260929 20:57:42.884613 18510 ts_tablet_manager.cc:1397] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20260929 20:57:42.885718 18508 raft_consensus.cc:348] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e680426e92482c90ef7e462568e123" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } } peers { permanent_uuid: "93131799a2b0463a9d2bca2c9f0d5f7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42785 } } peers { permanent_uuid: "d2f55ec0c2164972938745491569301e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36585 } }
I20260929 20:57:42.886246 18508 raft_consensus.cc:374] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:42.886343 18508 raft_consensus.cc:729] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93131799a2b0463a9d2bca2c9f0d5f7a, State: Initialized, Role: FOLLOWER
I20260929 20:57:42.887024 18508 consensus_queue.cc:260] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e680426e92482c90ef7e462568e123" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } } peers { permanent_uuid: "93131799a2b0463a9d2bca2c9f0d5f7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42785 } } peers { permanent_uuid: "d2f55ec0c2164972938745491569301e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36585 } }
I20260929 20:57:42.888226 18359 heartbeater.cc:503] Master 127.0.97.62:39737 was elected leader, sending a full tablet report...
I20260929 20:57:42.890681 18508 ts_tablet_manager.cc:1428] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a: Time spent starting tablet: real 0.019s user 0.011s sys 0.004s
I20260929 20:57:42.890970 18489 heartbeater.cc:503] Master 127.0.97.62:39737 was elected leader, sending a full tablet report...
I20260929 20:57:42.891628 18507 ts_tablet_manager.cc:1428] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123: Time spent starting tablet: real 0.024s user 0.016s sys 0.008s
I20260929 20:57:42.897929 18510 raft_consensus.cc:348] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e680426e92482c90ef7e462568e123" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } } peers { permanent_uuid: "93131799a2b0463a9d2bca2c9f0d5f7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42785 } } peers { permanent_uuid: "d2f55ec0c2164972938745491569301e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36585 } }
I20260929 20:57:42.898473 18510 raft_consensus.cc:374] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:42.898569 18510 raft_consensus.cc:729] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f55ec0c2164972938745491569301e, State: Initialized, Role: FOLLOWER
I20260929 20:57:42.899211 18510 consensus_queue.cc:260] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e680426e92482c90ef7e462568e123" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } } peers { permanent_uuid: "93131799a2b0463a9d2bca2c9f0d5f7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42785 } } peers { permanent_uuid: "d2f55ec0c2164972938745491569301e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36585 } }
I20260929 20:57:42.900241 18228 heartbeater.cc:503] Master 127.0.97.62:39737 was elected leader, sending a full tablet report...
I20260929 20:57:42.900903 18510 ts_tablet_manager.cc:1428] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
W20260929 20:57:43.016713 18490 tablet.cc:2369] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:57:43.059697 18514 raft_consensus.cc:482] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:57:43.060046 18514 raft_consensus.cc:504] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e680426e92482c90ef7e462568e123" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } } peers { permanent_uuid: "93131799a2b0463a9d2bca2c9f0d5f7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42785 } } peers { permanent_uuid: "d2f55ec0c2164972938745491569301e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36585 } }
I20260929 20:57:43.061800 18514 leader_election.cc:288] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8e680426e92482c90ef7e462568e123 (127.0.97.3:34065), d2f55ec0c2164972938745491569301e (127.0.97.1:36585)
I20260929 20:57:43.068357 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:43.062238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39195 (local address 127.0.97.3:34065)
0929 20:57:43.062447 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:43.062452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:43.063010 (+ 558us) server_negotiation.cc:407] Connection header received
0929 20:57:43.063267 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:43.063275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:43.063360 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:43.063423 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:43.064070 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:43.064785 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:43.066599 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:43.067447 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:43.067562 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:43.067660 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:43.067926 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 20:57:43.068047 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:57:43.068483 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:43.062170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34065 (local address 127.0.97.2:39195)
0929 20:57:43.062874 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:43.062894 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:43.063043 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:43.063618 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:43.063622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:43.063645 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:43.063935 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:43.063942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:43.064912 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:43.064916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:43.066400 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:43.066412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:43.067625 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:43.067631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:43.067777 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:43.067801 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:57:43.067880 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 20:57:43.068017 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":355,"threads_started":1}
I20260929 20:57:43.068820 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:43.063333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33131 (local address 127.0.97.1:36585)
0929 20:57:43.063544 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:43.063550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:43.063715 (+ 165us) server_negotiation.cc:407] Connection header received
0929 20:57:43.063928 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:43.063933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:43.064007 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:43.064082 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:43.064601 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:43.065269 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:43.066964 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:43.067837 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:43.067954 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:43.068042 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:43.068322 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 20:57:43.068461 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:57:43.068939 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:43.063047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36585 (local address 127.0.97.2:33131)
0929 20:57:43.063613 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:43.063628 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:43.063795 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:43.064111 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:43.064115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:43.064138 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:43.064452 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:43.064461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:43.065403 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:43.065409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:43.066782 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:43.066794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:43.068021 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:43.068027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:43.068174 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:43.068197 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:57:43.068279 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 20:57:43.068404 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":258,"threads_started":1}
I20260929 20:57:43.069434 18444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e1c2ab2c874e60ad3d3e2d0624e806" candidate_uuid: "93131799a2b0463a9d2bca2c9f0d5f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8e680426e92482c90ef7e462568e123" is_pre_election: true
I20260929 20:57:43.069953 18444 raft_consensus.cc:2454] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93131799a2b0463a9d2bca2c9f0d5f7a in term 0.
I20260929 20:57:43.070664 18249 leader_election.cc:302] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93131799a2b0463a9d2bca2c9f0d5f7a, c8e680426e92482c90ef7e462568e123; no voters:
I20260929 20:57:43.070766 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e1c2ab2c874e60ad3d3e2d0624e806" candidate_uuid: "93131799a2b0463a9d2bca2c9f0d5f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2f55ec0c2164972938745491569301e" is_pre_election: true
I20260929 20:57:43.071166 18514 raft_consensus.cc:2781] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:57:43.071311 18514 raft_consensus.cc:482] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:57:43.071404 18514 raft_consensus.cc:3037] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:43.071338 18183 raft_consensus.cc:2454] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93131799a2b0463a9d2bca2c9f0d5f7a in term 0.
I20260929 20:57:43.073800 18514 raft_consensus.cc:504] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e680426e92482c90ef7e462568e123" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } } peers { permanent_uuid: "93131799a2b0463a9d2bca2c9f0d5f7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42785 } } peers { permanent_uuid: "d2f55ec0c2164972938745491569301e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36585 } }
I20260929 20:57:43.074467 18514 leader_election.cc:288] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [CANDIDATE]: Term 1 election: Requested vote from peers c8e680426e92482c90ef7e462568e123 (127.0.97.3:34065), d2f55ec0c2164972938745491569301e (127.0.97.1:36585)
I20260929 20:57:43.074999 18444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e1c2ab2c874e60ad3d3e2d0624e806" candidate_uuid: "93131799a2b0463a9d2bca2c9f0d5f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8e680426e92482c90ef7e462568e123"
I20260929 20:57:43.075084 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e1c2ab2c874e60ad3d3e2d0624e806" candidate_uuid: "93131799a2b0463a9d2bca2c9f0d5f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2f55ec0c2164972938745491569301e"
I20260929 20:57:43.075212 18444 raft_consensus.cc:3037] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:43.075300 18183 raft_consensus.cc:3037] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:43.077554 18183 raft_consensus.cc:2454] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93131799a2b0463a9d2bca2c9f0d5f7a in term 1.
I20260929 20:57:43.077564 18444 raft_consensus.cc:2454] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93131799a2b0463a9d2bca2c9f0d5f7a in term 1.
I20260929 20:57:43.077900 18249 leader_election.cc:302] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93131799a2b0463a9d2bca2c9f0d5f7a, c8e680426e92482c90ef7e462568e123; no voters:
I20260929 20:57:43.078148 18514 raft_consensus.cc:2781] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:57:43.078644 18514 raft_consensus.cc:686] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [term 1 LEADER]: Becoming Leader. State: Replica: 93131799a2b0463a9d2bca2c9f0d5f7a, State: Running, Role: LEADER
I20260929 20:57:43.079154 18514 consensus_queue.cc:237] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e680426e92482c90ef7e462568e123" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } } peers { permanent_uuid: "93131799a2b0463a9d2bca2c9f0d5f7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42785 } } peers { permanent_uuid: "d2f55ec0c2164972938745491569301e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36585 } }
W20260929 20:57:43.081563 18360 tablet.cc:2369] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:57:43.084092 18042 catalog_manager.cc:5495] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 93131799a2b0463a9d2bca2c9f0d5f7a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "93131799a2b0463a9d2bca2c9f0d5f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e680426e92482c90ef7e462568e123" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93131799a2b0463a9d2bca2c9f0d5f7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2f55ec0c2164972938745491569301e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36585 } health_report { overall_health: UNKNOWN } } }
W20260929 20:57:43.146221 18229 tablet.cc:2369] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:57:43.150243 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:43.144599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40298 (local address 127.0.97.2:42785)
0929 20:57:43.144830 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:43.144835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:43.144852 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:57:43.145068 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:43.145073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:43.145156 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:43.145294 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:57:43.145667 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:43.146675 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:43.148122 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:43.148526 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:43.148623 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:43.149147 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:57:43.149628 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:57:43.149734 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:43.149903 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 20:57:43.149971 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:57:43.158454 18444 raft_consensus.cc:1261] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123 [term 1 FOLLOWER]: Refusing update from remote peer 93131799a2b0463a9d2bca2c9f0d5f7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:57:43.158454 18183 raft_consensus.cc:1261] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e [term 1 FOLLOWER]: Refusing update from remote peer 93131799a2b0463a9d2bca2c9f0d5f7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:57:43.159528 18518 consensus_queue.cc:1035] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8e680426e92482c90ef7e462568e123" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:57:43.159868 18514 consensus_queue.cc:1035] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2f55ec0c2164972938745491569301e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:57:43.170418 18521 mvcc.cc:205] Tried to move back new op lower bound from 7334770537086218240 to 7334770536776491008. Current Snapshot: MvccSnapshot[applied={T|T < 7334770537086218240}]
I20260929 20:57:43.170600 18522 mvcc.cc:205] Tried to move back new op lower bound from 7334770537086218240 to 7334770536776491008. Current Snapshot: MvccSnapshot[applied={T|T < 7334770537086218240}]
I20260929 20:57:43.172760 18523 mvcc.cc:205] Tried to move back new op lower bound from 7334770537086218240 to 7334770536776491008. Current Snapshot: MvccSnapshot[applied={T|T < 7334770537086218240}]
I20260929 20:57:43.183615 18042 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:57:43.183892 18042 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:57:43.187474 18042 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1241cfc8d30f4bb7890d3c9c86449184: Sending DeleteTablet for 3 replicas of tablet 91e1c2ab2c874e60ad3d3e2d0624e806
I20260929 20:57:43.188686 18294 tablet_service.cc:1514] Processing DeleteTablet for tablet 91e1c2ab2c874e60ad3d3e2d0624e806 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:57:43 UTC) from {username='slave'} at 127.0.0.1:40288
I20260929 20:57:43.188694 18424 tablet_service.cc:1514] Processing DeleteTablet for tablet 91e1c2ab2c874e60ad3d3e2d0624e806 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:57:43 UTC) from {username='slave'} at 127.0.0.1:55386
I20260929 20:57:43.188916 18163 tablet_service.cc:1514] Processing DeleteTablet for tablet 91e1c2ab2c874e60ad3d3e2d0624e806 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:57:43 UTC) from {username='slave'} at 127.0.0.1:43172
I20260929 20:57:43.191303 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 18101
I20260929 20:57:43.211326 18532 tablet_replica.cc:330] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e: stopping tablet replica
I20260929 20:57:43.211920 18532 raft_consensus.cc:2229] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:57:43.212484 18532 raft_consensus.cc:2258] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:57:43.215240 18532 ts_tablet_manager.cc:1905] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:57:43.215401 18530 tablet_replica.cc:330] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a: stopping tablet replica
I20260929 20:57:43.215860 18530 raft_consensus.cc:2229] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:57:43.216518 18530 raft_consensus.cc:2258] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:57:43.217869 18531 tablet_replica.cc:330] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123: stopping tablet replica
I20260929 20:57:43.218333 18531 raft_consensus.cc:2229] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:57:43.218696 18530 ts_tablet_manager.cc:1905] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:57:43.218814 18531 raft_consensus.cc:2258] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:57:43.220947 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:43.192663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43182 (local address 127.0.97.1:36585)
0929 20:57:43.192861 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:43.192868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:43.210084 (+ 17216us) server_negotiation.cc:407] Connection header received
0929 20:57:43.210961 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:43.210968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:43.211088 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:43.211199 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:43.213359 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:43.214631 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:43.216866 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:43.217468 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:43.217572 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:43.219246 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:43.219258 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:43.219275 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:43.220024 (+ 749us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:43.220085 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:43.220111 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:43.220193 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:43.220427 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 20:57:43.220551 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:57:43.222391 18223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:43.225411 18531 ts_tablet_manager.cc:1905] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:57:43.228509 18530 ts_tablet_manager.cc:1918] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:57:43.228681 18530 log.cc:1196] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/91e1c2ab2c874e60ad3d3e2d0624e806
I20260929 20:57:43.229228 18530 ts_tablet_manager.cc:1939] T 91e1c2ab2c874e60ad3d3e2d0624e806 P 93131799a2b0463a9d2bca2c9f0d5f7a: Deleting consensus metadata
I20260929 20:57:43.231251 18029 catalog_manager.cc:4841] TS 93131799a2b0463a9d2bca2c9f0d5f7a (127.0.97.2:42785): tablet 91e1c2ab2c874e60ad3d3e2d0624e806 (table test-table [id=4323cefcff1d4abdab2391bdcc36e0fe]) successfully deleted
I20260929 20:57:43.232241 18531 ts_tablet_manager.cc:1918] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:57:43.232406 18531 log.cc:1196] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/91e1c2ab2c874e60ad3d3e2d0624e806
I20260929 20:57:43.232910 18531 ts_tablet_manager.cc:1939] T 91e1c2ab2c874e60ad3d3e2d0624e806 P c8e680426e92482c90ef7e462568e123: Deleting consensus metadata
I20260929 20:57:43.234761 18028 catalog_manager.cc:4841] TS c8e680426e92482c90ef7e462568e123 (127.0.97.3:34065): tablet 91e1c2ab2c874e60ad3d3e2d0624e806 (table test-table [id=4323cefcff1d4abdab2391bdcc36e0fe]) successfully deleted
I20260929 20:57:43.318657 18532 ts_tablet_manager.cc:1918] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:57:43.318970 18532 log.cc:1196] T 91e1c2ab2c874e60ad3d3e2d0624e806 P d2f55ec0c2164972938745491569301e: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/91e1c2ab2c874e60ad3d3e2d0624e806
I20260929 20:57:43.319305 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 18101
W20260929 20:57:43.341511 18028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:36585 (error 108)
I20260929 20:57:43.342660 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 18232
I20260929 20:57:43.353478 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:43.344201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40304 (local address 127.0.97.2:42785)
0929 20:57:43.344403 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:43.344409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:43.344428 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:57:43.344488 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:43.344493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:43.344596 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:43.344690 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:43.345459 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:43.346660 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:43.348586 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:43.349255 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:43.349374 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:43.350747 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:43.350759 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:43.350778 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:43.351910 (+ 1132us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:43.351973 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:43.352007 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:43.352118 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:43.352956 (+ 838us) server_negotiation.cc:299] Negotiation successful
0929 20:57:43.353060 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 20:57:43.355059 18354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:43.359052 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:43.357594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36585 (local address 127.0.0.1:43186)
0929 20:57:43.358565 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:43.358767 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":838,"thread_start_us":714,"threads_started":1}
W20260929 20:57:43.359365 18028 catalog_manager.cc:4584] TS d2f55ec0c2164972938745491569301e (127.0.97.1:36585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91e1c2ab2c874e60ad3d3e2d0624e806: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36585: connect: Connection refused (error 111)
I20260929 20:57:43.413055 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:43.412518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36585 (local address 127.0.0.1:43196)
0929 20:57:43.412759 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:43.412873 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 20:57:43.414131 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:43.413732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36585 (local address 127.0.0.1:43212)
0929 20:57:43.413885 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:43.413970 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 20:57:43.467665 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 18232
I20260929 20:57:43.474735 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:43.474131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36585 (local address 127.0.0.1:43222)
0929 20:57:43.474410 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:43.474533 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 20:57:43.475867 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:43.475471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36585 (local address 127.0.0.1:43238)
0929 20:57:43.475617 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:43.475701 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 20:57:43.484364 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 18362
I20260929 20:57:43.490739 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:43.485188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55398 (local address 127.0.97.3:34065)
0929 20:57:43.485362 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:43.485366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:43.485385 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:57:43.485567 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:43.485572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:43.485670 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:43.485760 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:43.486206 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:43.486937 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:43.488197 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:43.488567 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:43.488656 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:43.489539 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:43.489548 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:43.489563 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:43.490111 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:43.490161 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:43.490181 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:43.490304 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:43.490397 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 20:57:43.490482 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:57:43.491374 18484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:43.556869 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:43.556353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36585 (local address 127.0.0.1:43244)
0929 20:57:43.556605 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:43.556704 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 20:57:43.558039 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:43.557592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36585 (local address 127.0.0.1:43248)
0929 20:57:43.557828 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:43.557905 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260929 20:57:43.579035 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 18362
I20260929 20:57:43.592432 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 18011
I20260929 20:57:43.593684 18072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:43.702679 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 18011
2026-09-29T20:57:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2844 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:57:43 dist-test-slave-m9g1 krb5kdc[18542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:57:43 dist-test-slave-m9g1 krb5kdc[18542](info): set up 2 sockets
Sep 29 20:57:43 dist-test-slave-m9g1 krb5kdc[18542](info): commencing operation
krb5kdc: starting...
W20260929 20:57:44.726052 388 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:57:44 dist-test-slave-m9g1 krb5kdc[18542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:57:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:57:44.959848 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44029
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:57:45.280117 18555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:45.280428 18555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:45.280572 18555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:45.287755 18555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:57:45.287878 18555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:45.287951 18555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:45.288130 18555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:57:45.288215 18555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:57:45.298120 18555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:45.299463 18555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:45.300788 18555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:45.312536 18561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:45.312493 18560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:45.312486 18563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:45.313387 18555 server_base.cc:1034] running on GCE node
I20260929 20:57:45.314127 18555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:45.315539 18555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:45.316757 18555 hybrid_clock.cc:648] HybridClock initialized: now 1790715465316700 us; error 49 us; skew 500 ppm
I20260929 20:57:45.317119 18555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:57:45 dist-test-slave-m9g1 krb5kdc[18542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:57:45.320230 18555 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:57:45.321743 18555 webserver.cc:460] Webserver started at http://127.0.97.62:44633/ using document root <none> and password file <none>
I20260929 20:57:45.322508 18555 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:45.322634 18555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:45.322947 18555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:45.326138 18555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "50de1f83e349487c98eb41909c81302e"
format_stamp: "Formatted at 2026-09-29 20:57:45 on dist-test-slave-m9g1"
server_key: "f1e526a5a1c6fb310b82da2228574ece"
server_key_iv: "f71a69896875bc341804c57c93b54672"
server_key_version: "encryptionkey@0"
I20260929 20:57:45.326982 18555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "50de1f83e349487c98eb41909c81302e"
format_stamp: "Formatted at 2026-09-29 20:57:45 on dist-test-slave-m9g1"
server_key: "f1e526a5a1c6fb310b82da2228574ece"
server_key_iv: "f71a69896875bc341804c57c93b54672"
server_key_version: "encryptionkey@0"
I20260929 20:57:45.332208 18555 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:57:45.335800 18570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:45.337237 18555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:57:45.337409 18555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "50de1f83e349487c98eb41909c81302e"
format_stamp: "Formatted at 2026-09-29 20:57:45 on dist-test-slave-m9g1"
server_key: "f1e526a5a1c6fb310b82da2228574ece"
server_key_iv: "f71a69896875bc341804c57c93b54672"
server_key_version: "encryptionkey@0"
I20260929 20:57:45.337589 18555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:45.361542 18555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:45.365516 18555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:45.365768 18555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:45.383059 18555 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44029
I20260929 20:57:45.383081 18621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44029 every 8 connection(s)
I20260929 20:57:45.384385 18555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:57:45.388080 18622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:45.391950 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 18555
I20260929 20:57:45.392130 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:57:45.392665 388 external_mini_cluster.cc:1428] Setting key dbcf0c8f8becd11b21a8f008027d64e4
I20260929 20:57:45.401108 18622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e: Bootstrap starting.
I20260929 20:57:45.404863 18622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:45.406162 18622 log.cc:824] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:57:45 dist-test-slave-m9g1 krb5kdc[18542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:57:45.409349 18622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e: No bootstrap required, opened a new log
I20260929 20:57:45.415889 18625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:45.394637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.0.97.62:44029)
0929 20:57:45.395404 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:45.395416 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:45.395457 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:57:45.397043 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:45.397066 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:45.397488 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:45.397816 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:45.399542 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:45.400861 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:45.402737 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:45.403549 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:45.403682 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:45.408525 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:45.408537 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:45.408557 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:45.408600 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:45.411742 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:45.412445 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:45.412455 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:45.412462 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:45.412638 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:45.413034 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:45.413038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:45.413040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:45.413606 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:45.413897 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:45.414442 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 20:57:45.414858 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":360,"threads_started":1}
I20260929 20:57:45.417126 18622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50de1f83e349487c98eb41909c81302e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44029 } }
I20260929 20:57:45.417486 18622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:45.417577 18622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50de1f83e349487c98eb41909c81302e, State: Initialized, Role: FOLLOWER
I20260929 20:57:45.418154 18622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50de1f83e349487c98eb41909c81302e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44029 } }
I20260929 20:57:45.418447 18622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:57:45.418630 18622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:57:45.418814 18622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:45.420929 18622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50de1f83e349487c98eb41909c81302e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44029 } }
I20260929 20:57:45.421377 18622 leader_election.cc:302] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50de1f83e349487c98eb41909c81302e; no voters:
I20260929 20:57:45.421947 18622 leader_election.cc:288] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:57:45.422098 18627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:57:45.423946 18622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:57:45.424350 18627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e [term 1 LEADER]: Becoming Leader. State: Replica: 50de1f83e349487c98eb41909c81302e, State: Running, Role: LEADER
I20260929 20:57:45.424855 18627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50de1f83e349487c98eb41909c81302e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44029 } }
I20260929 20:57:45.427515 18629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50de1f83e349487c98eb41909c81302e. Latest consensus state: current_term: 1 leader_uuid: "50de1f83e349487c98eb41909c81302e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50de1f83e349487c98eb41909c81302e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44029 } } }
I20260929 20:57:45.427922 18629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e [sys.catalog]: This master's current role is: LEADER
I20260929 20:57:45.428066 18628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50de1f83e349487c98eb41909c81302e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50de1f83e349487c98eb41909c81302e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44029 } } }
I20260929 20:57:45.428267 18628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e [sys.catalog]: This master's current role is: LEADER
I20260929 20:57:45.429023 18639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:57:45.434855 18639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:57:45.452376 18639 catalog_manager.cc:1347] Generated new cluster ID: 2a5fdeb32c734f0392730d458ead84a5
I20260929 20:57:45.452507 18639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:57:45.472354 18639 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:57:45.474335 18639 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:57:45.482338 18639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e: Generated new TSK 0
I20260929 20:57:45.483377 18639 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:57:45.553063 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44029
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:57:45.877698 18648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:45.878007 18648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:45.878101 18648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:45.885396 18648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:45.885509 18648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:45.885622 18648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:57:45.895406 18648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:45.896624 18648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:45.897842 18648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:45.910243 18653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:45.910269 18654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:45.910319 18656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:45.911000 18648 server_base.cc:1034] running on GCE node
I20260929 20:57:45.911531 18648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:45.912387 18648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:45.913586 18648 hybrid_clock.cc:648] HybridClock initialized: now 1790715465913531 us; error 45 us; skew 500 ppm
I20260929 20:57:45.913908 18648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:57:45 dist-test-slave-m9g1 krb5kdc[18542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:57:45.916899 18648 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:57:45.918375 18648 webserver.cc:460] Webserver started at http://127.0.97.1:37391/ using document root <none> and password file <none>
I20260929 20:57:45.919009 18648 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:45.919101 18648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:45.919346 18648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:45.922099 18648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "4024b06066bb48bc9586205d0cdb9f10"
format_stamp: "Formatted at 2026-09-29 20:57:45 on dist-test-slave-m9g1"
server_key: "9b06ae4eb68d470580ef4a4eac992a78"
server_key_iv: "f4213ed4b1987373ef8b915d76ceb735"
server_key_version: "encryptionkey@0"
I20260929 20:57:45.922870 18648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4024b06066bb48bc9586205d0cdb9f10"
format_stamp: "Formatted at 2026-09-29 20:57:45 on dist-test-slave-m9g1"
server_key: "9b06ae4eb68d470580ef4a4eac992a78"
server_key_iv: "f4213ed4b1987373ef8b915d76ceb735"
server_key_version: "encryptionkey@0"
I20260929 20:57:45.927629 18648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:57:45.930908 18663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:45.932184 18648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:57:45.932305 18648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "4024b06066bb48bc9586205d0cdb9f10"
format_stamp: "Formatted at 2026-09-29 20:57:45 on dist-test-slave-m9g1"
server_key: "9b06ae4eb68d470580ef4a4eac992a78"
server_key_iv: "f4213ed4b1987373ef8b915d76ceb735"
server_key_version: "encryptionkey@0"
I20260929 20:57:45.932459 18648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:45.945590 18648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:45.949606 18648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:45.949868 18648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:45.950814 18648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:45.952085 18648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:45.952155 18648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:45.952244 18648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:45.952288 18648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:45.981185 18648 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32937
I20260929 20:57:45.981261 18775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32937 every 8 connection(s)
I20260929 20:57:45.982477 18648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:57:45.989482 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 18648
I20260929 20:57:45.989622 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:57:45.990043 388 external_mini_cluster.cc:1428] Setting key b12c84649ca76d2faac5606486b30052
Sep 29 20:57:45 dist-test-slave-m9g1 krb5kdc[18542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:57:46.003391 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:45.985165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39775 (local address 127.0.97.62:44029)
0929 20:57:45.985552 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:45.985557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:45.986640 (+ 1083us) server_negotiation.cc:407] Connection header received
0929 20:57:45.988408 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:45.988413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:45.988479 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:45.988576 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:45.990163 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:45.991114 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:45.993023 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:45.993633 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:45.993757 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:45.997286 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:45.997292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:45.997297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:45.997348 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:46.000236 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:46.001033 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:46.001037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:46.001040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:46.001177 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:46.001641 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:46.001644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:46.001646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:46.001934 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:46.002045 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:46.002946 (+ 901us) server_negotiation.cc:299] Negotiation successful
0929 20:57:46.003131 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":193,"threads_started":1}
I20260929 20:57:46.004702 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:45.985718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44029 (local address 127.0.97.1:39775)
0929 20:57:45.986485 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:45.986526 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:45.988150 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:45.988787 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:45.988797 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:45.989336 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:57:45.989978 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:45.989998 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:45.991272 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:45.991277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:45.992875 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:45.992889 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:45.993827 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:45.993831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:45.994043 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:45.994784 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:57:45.994809 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:57:45.997032 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:57:46.000402 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:57:46.000412 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:57:46.000428 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:57:46.000891 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:57:46.001283 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:57:46.001288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:57:46.001292 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:57:46.001540 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:57:46.002047 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:57:46.002055 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:57:46.002523 (+ 468us) client_negotiation.cc:769] Sending connection context
0929 20:57:46.002846 (+ 323us) client_negotiation.cc:240] Negotiation successful
0929 20:57:46.003377 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":264,"threads_started":1}
I20260929 20:57:46.006223 18776 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44029
I20260929 20:57:46.006556 18776 heartbeater.cc:464] Registering TS with master...
I20260929 20:57:46.007407 18776 heartbeater.cc:511] Master 127.0.97.62:44029 requested a full tablet report, sending...
I20260929 20:57:46.009521 18587 ts_manager.cc:194] Registered new tserver with Master: 4024b06066bb48bc9586205d0cdb9f10 (127.0.97.1:32937)
I20260929 20:57:46.010986 18587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39775
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:57:46.045521 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44029
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:57:46.364125 18782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:46.364421 18782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:46.364544 18782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:46.371908 18782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:46.372017 18782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:46.372164 18782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:57:46.381793 18782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:46.383116 18782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:46.384358 18782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:46.397760 18787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:46.397742 18788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:46.398142 18790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:46.399276 18782 server_base.cc:1034] running on GCE node
I20260929 20:57:46.399840 18782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:46.400753 18782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:46.401948 18782 hybrid_clock.cc:648] HybridClock initialized: now 1790715466401899 us; error 44 us; skew 500 ppm
I20260929 20:57:46.402285 18782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:57:46 dist-test-slave-m9g1 krb5kdc[18542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:57:46.405488 18782 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:57:46.406888 18782 webserver.cc:460] Webserver started at http://127.0.97.2:46543/ using document root <none> and password file <none>
I20260929 20:57:46.407585 18782 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:46.407696 18782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:46.407974 18782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:46.410780 18782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "f857491087b6409b8b607eddc47f6152"
format_stamp: "Formatted at 2026-09-29 20:57:46 on dist-test-slave-m9g1"
server_key: "b1b447ac6f4dfa1da193a669c53d71a8"
server_key_iv: "c20b3c4c2869825b5aa981030da93319"
server_key_version: "encryptionkey@0"
I20260929 20:57:46.411520 18782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f857491087b6409b8b607eddc47f6152"
format_stamp: "Formatted at 2026-09-29 20:57:46 on dist-test-slave-m9g1"
server_key: "b1b447ac6f4dfa1da193a669c53d71a8"
server_key_iv: "c20b3c4c2869825b5aa981030da93319"
server_key_version: "encryptionkey@0"
I20260929 20:57:46.416237 18782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:57:46.419533 18797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:46.420791 18782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:57:46.420938 18782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "f857491087b6409b8b607eddc47f6152"
format_stamp: "Formatted at 2026-09-29 20:57:46 on dist-test-slave-m9g1"
server_key: "b1b447ac6f4dfa1da193a669c53d71a8"
server_key_iv: "c20b3c4c2869825b5aa981030da93319"
server_key_version: "encryptionkey@0"
I20260929 20:57:46.421108 18782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:46.436632 18782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:46.440656 18782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:46.440927 18782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:46.441866 18782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:46.443270 18782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:46.443337 18782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:46.443452 18782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:46.443504 18782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:46.472399 18782 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34463
I20260929 20:57:46.472469 18909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34463 every 8 connection(s)
I20260929 20:57:46.473609 18782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:57:46.478505 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 18782
I20260929 20:57:46.478658 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:57:46.479069 388 external_mini_cluster.cc:1428] Setting key 9b9e6d864567d0378bb98c43ef175b82
Sep 29 20:57:46 dist-test-slave-m9g1 krb5kdc[18542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:57:46.494961 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:46.476051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52199 (local address 127.0.97.62:44029)
0929 20:57:46.476256 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:46.476260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:46.477342 (+ 1082us) server_negotiation.cc:407] Connection header received
0929 20:57:46.479195 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:46.479201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:46.479298 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:46.479422 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:46.481035 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:46.481760 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:46.483862 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:46.484325 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:46.484422 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:46.488423 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:46.488429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:46.488433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:46.488471 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:46.491544 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:46.492471 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:46.492476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:46.492478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:46.492622 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:46.493109 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:46.493113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:46.493115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:46.493407 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:46.493545 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:46.494345 (+ 800us) server_negotiation.cc:299] Negotiation successful
0929 20:57:46.494606 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 20:57:46.496016 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:46.476515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44029 (local address 127.0.97.2:52199)
0929 20:57:46.477191 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:46.477227 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:46.478940 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:46.479598 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:46.479606 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:46.480148 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:57:46.480891 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:46.480910 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:46.481910 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:46.481915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:46.483710 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:46.483723 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:46.484521 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:46.484527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:46.484778 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:46.485426 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:57:46.485452 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:57:46.488127 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:57:46.491748 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:57:46.491756 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:57:46.491785 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:57:46.492280 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:57:46.492746 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:57:46.492750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:57:46.492753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:57:46.492976 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:57:46.493541 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:57:46.493550 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:57:46.493933 (+ 383us) client_negotiation.cc:769] Sending connection context
0929 20:57:46.494213 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 20:57:46.494802 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":276,"threads_started":1}
I20260929 20:57:46.497977 18910 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44029
I20260929 20:57:46.498369 18910 heartbeater.cc:464] Registering TS with master...
I20260929 20:57:46.499161 18910 heartbeater.cc:511] Master 127.0.97.62:44029 requested a full tablet report, sending...
I20260929 20:57:46.500795 18587 ts_manager.cc:194] Registered new tserver with Master: f857491087b6409b8b607eddc47f6152 (127.0.97.2:34463)
I20260929 20:57:46.501943 18587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52199
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:57:46.532656 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44029
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:57:46.850178 18915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:46.850498 18915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:46.850629 18915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:46.857789 18915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:46.857905 18915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:46.858060 18915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:57:46.867672 18915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:46.868898 18915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:46.870221 18915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:46.882995 18921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:46.883059 18923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:46.883036 18920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:46.884177 18915 server_base.cc:1034] running on GCE node
I20260929 20:57:46.884744 18915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:46.885643 18915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:46.886873 18915 hybrid_clock.cc:648] HybridClock initialized: now 1790715466886785 us; error 50 us; skew 500 ppm
I20260929 20:57:46.887181 18915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:57:46 dist-test-slave-m9g1 krb5kdc[18542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:57:46.894347 18915 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:57:46.895741 18915 webserver.cc:460] Webserver started at http://127.0.97.3:36579/ using document root <none> and password file <none>
I20260929 20:57:46.896497 18915 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:46.896610 18915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:46.896915 18915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:46.899873 18915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "04d5f0b685c3461cbd4286c321aa0112"
format_stamp: "Formatted at 2026-09-29 20:57:46 on dist-test-slave-m9g1"
server_key: "51ea4029970abfc60e39e8fe370fd93d"
server_key_iv: "6e844f482389ceff2731865c83d26ff1"
server_key_version: "encryptionkey@0"
I20260929 20:57:46.900687 18915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "04d5f0b685c3461cbd4286c321aa0112"
format_stamp: "Formatted at 2026-09-29 20:57:46 on dist-test-slave-m9g1"
server_key: "51ea4029970abfc60e39e8fe370fd93d"
server_key_iv: "6e844f482389ceff2731865c83d26ff1"
server_key_version: "encryptionkey@0"
I20260929 20:57:46.906035 18915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 20:57:46.909539 18930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:46.910943 18915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:57:46.911087 18915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "04d5f0b685c3461cbd4286c321aa0112"
format_stamp: "Formatted at 2026-09-29 20:57:46 on dist-test-slave-m9g1"
server_key: "51ea4029970abfc60e39e8fe370fd93d"
server_key_iv: "6e844f482389ceff2731865c83d26ff1"
server_key_version: "encryptionkey@0"
I20260929 20:57:46.911279 18915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:46.924788 18915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:46.929131 18915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:46.929394 18915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:46.930375 18915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:46.931685 18915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:46.931756 18915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:46.931898 18915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:46.931954 18915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:46.962066 18915 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42769
I20260929 20:57:46.962103 19042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42769 every 8 connection(s)
I20260929 20:57:46.963366 18915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:57:46.964986 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 18915
I20260929 20:57:46.965149 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:57:46.965682 388 external_mini_cluster.cc:1428] Setting key 7bc06a03bd2095ec2413c2d41d25f317
Sep 29 20:57:46 dist-test-slave-m9g1 krb5kdc[18542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:57:46.983503 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:46.965998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54635 (local address 127.0.97.62:44029)
0929 20:57:46.966203 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:46.966209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:46.967552 (+ 1343us) server_negotiation.cc:407] Connection header received
0929 20:57:46.969387 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:46.969391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:46.969455 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:46.969533 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:46.971088 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:46.971879 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:46.973424 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:46.973816 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:46.973921 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:46.977429 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:46.977434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:46.977438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:46.977477 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:46.980300 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:46.981191 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:46.981195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:46.981198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:46.981336 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:46.981813 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:46.981815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:46.981817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:46.982102 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:46.982257 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:46.983077 (+ 820us) server_negotiation.cc:299] Negotiation successful
0929 20:57:46.983252 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:57:46.984797 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:46.966601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44029 (local address 127.0.97.3:54635)
0929 20:57:46.967403 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:46.967457 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:46.969130 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:46.969729 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:46.969737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:46.970343 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:57:46.970938 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:46.970957 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:46.972016 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:46.972020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:46.973284 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:46.973296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:46.974015 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:46.974020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:46.974268 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:46.974943 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:57:46.974966 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:57:46.977198 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:57:46.980451 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:57:46.980461 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:57:46.980480 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:57:46.981057 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:57:46.981438 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:57:46.981443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:57:46.981446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:57:46.981690 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:57:46.982243 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:57:46.982251 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:57:46.982667 (+ 416us) client_negotiation.cc:769] Sending connection context
0929 20:57:46.982955 (+ 288us) client_negotiation.cc:240] Negotiation successful
0929 20:57:46.983486 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":323,"threads_started":1}
I20260929 20:57:46.986150 19043 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44029
I20260929 20:57:46.986491 19043 heartbeater.cc:464] Registering TS with master...
I20260929 20:57:46.987161 19043 heartbeater.cc:511] Master 127.0.97.62:44029 requested a full tablet report, sending...
I20260929 20:57:46.988619 18587 ts_manager.cc:194] Registered new tserver with Master: 04d5f0b685c3461cbd4286c321aa0112 (127.0.97.3:42769)
I20260929 20:57:46.989431 18587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54635
I20260929 20:57:46.992992 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:57:47.010228 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:46.997134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59522 (local address 127.0.97.62:44029)
0929 20:57:46.997341 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:46.997345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:46.997720 (+ 375us) server_negotiation.cc:407] Connection header received
0929 20:57:46.997875 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:46.997879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:46.997942 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:46.998033 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:46.999387 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.000129 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.001464 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.001913 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.002016 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:47.004881 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:47.004886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:47.004890 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:47.004953 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:47.007451 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:47.008272 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:47.008276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:47.008278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:47.008401 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:47.008925 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:47.008928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:47.008930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:47.009226 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:47.009417 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:47.009790 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 20:57:47.009961 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 20:57:47.014461 18776 heartbeater.cc:503] Master 127.0.97.62:44029 was elected leader, sending a full tablet report...
I20260929 20:57:47.014137 18587 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:57:47.021500 18587 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:57:47.044826 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.034618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49426 (local address 127.0.97.3:42769)
0929 20:57:47.035593 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:47.035604 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:47.036088 (+ 484us) server_negotiation.cc:407] Connection header received
0929 20:57:47.036380 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:47.036389 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:47.036680 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:47.036851 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:47.037697 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.039306 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.042036 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.043137 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.043250 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:47.043506 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:47.044319 (+ 813us) server_negotiation.cc:299] Negotiation successful
0929 20:57:47.044491 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":655,"threads_started":1}
I20260929 20:57:47.044866 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.034096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42769 (local address 127.0.0.1:49426)
0929 20:57:47.035980 (+ 1884us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:47.035996 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:47.036188 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:47.037064 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:47.037072 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:47.037102 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:47.037479 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:47.037493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.039466 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.039472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:47.041870 (+ 2398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:47.041882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.043641 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.043650 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:47.044076 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:47.044136 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 20:57:47.044258 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 20:57:47.044468 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1756,"mutex_wait_us":211,"thread_start_us":1690,"threads_started":1}
I20260929 20:57:47.046005 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.035170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32937 (local address 127.0.0.1:45648)
0929 20:57:47.037924 (+ 2754us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:47.037942 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:47.038060 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:47.038718 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:47.038722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:47.038748 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:47.039170 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:47.039187 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.040979 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.040983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:47.043037 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:47.043052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.045122 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.045128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:47.045286 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:47.045332 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 20:57:47.045422 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 20:57:47.045567 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2646,"mutex_wait_us":23,"thread_start_us":2521,"threads_started":1}
I20260929 20:57:47.046440 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.033987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34463 (local address 127.0.0.1:34320)
0929 20:57:47.034725 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:47.034771 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:47.034957 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:47.039107 (+ 4150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:47.039112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:47.039139 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:47.039573 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:47.039585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.041662 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.041667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:47.043786 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:47.043800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.045576 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.045581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:47.045749 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:47.045780 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:57:47.045846 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 20:57:47.045975 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":1,"thread_start_us":333,"threads_started":1}
I20260929 20:57:47.046866 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.034662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34320 (local address 127.0.97.2:34463)
0929 20:57:47.038555 (+ 3893us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:47.038563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:47.038589 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:57:47.038675 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:47.038681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:47.038904 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:47.039802 (+ 898us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:47.040019 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.041488 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.043975 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.045377 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.046115 (+ 738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:47.046329 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:47.046413 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 20:57:47.046591 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3726,"thread_start_us":3620,"threads_started":1}
I20260929 20:57:47.047184 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.036229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45648 (local address 127.0.97.1:32937)
0929 20:57:47.037544 (+ 1315us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:47.037552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:47.038057 (+ 505us) server_negotiation.cc:407] Connection header received
0929 20:57:47.038227 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:47.038237 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:47.038526 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:47.038723 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:47.039425 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.040824 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.043222 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.044630 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.044747 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:47.044980 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:47.046693 (+ 1713us) server_negotiation.cc:299] Negotiation successful
0929 20:57:47.046880 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1134,"thread_start_us":973,"threads_started":1}
I20260929 20:57:47.049156 18978 tablet_service.cc:1467] Processing CreateTablet for tablet 8af63a4acd194c6a9c496b3318069ac9 (DEFAULT_TABLE table=test-table [id=e0652337c09d4d2baaf4141b57fd74e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:57:47.049727 18845 tablet_service.cc:1467] Processing CreateTablet for tablet 8af63a4acd194c6a9c496b3318069ac9 (DEFAULT_TABLE table=test-table [id=e0652337c09d4d2baaf4141b57fd74e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:57:47.050796 18978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af63a4acd194c6a9c496b3318069ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:47.051146 18845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af63a4acd194c6a9c496b3318069ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:47.050752 18711 tablet_service.cc:1467] Processing CreateTablet for tablet 8af63a4acd194c6a9c496b3318069ac9 (DEFAULT_TABLE table=test-table [id=e0652337c09d4d2baaf4141b57fd74e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:57:47.052163 18711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af63a4acd194c6a9c496b3318069ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:47.063654 19061 tablet_bootstrap.cc:492] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152: Bootstrap starting.
I20260929 20:57:47.066435 19063 tablet_bootstrap.cc:492] T 8af63a4acd194c6a9c496b3318069ac9 P 4024b06066bb48bc9586205d0cdb9f10: Bootstrap starting.
I20260929 20:57:47.067309 19061 tablet_bootstrap.cc:654] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:47.068740 19061 log.cc:824] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:47.070154 19063 tablet_bootstrap.cc:654] T 8af63a4acd194c6a9c496b3318069ac9 P 4024b06066bb48bc9586205d0cdb9f10: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:47.071256 19062 tablet_bootstrap.cc:492] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112: Bootstrap starting.
I20260929 20:57:47.071693 19063 log.cc:824] T 8af63a4acd194c6a9c496b3318069ac9 P 4024b06066bb48bc9586205d0cdb9f10: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:47.074846 19062 tablet_bootstrap.cc:654] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:47.075137 19061 tablet_bootstrap.cc:492] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152: No bootstrap required, opened a new log
I20260929 20:57:47.075155 19063 tablet_bootstrap.cc:492] T 8af63a4acd194c6a9c496b3318069ac9 P 4024b06066bb48bc9586205d0cdb9f10: No bootstrap required, opened a new log
I20260929 20:57:47.075408 19061 ts_tablet_manager.cc:1397] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260929 20:57:47.075433 19063 ts_tablet_manager.cc:1397] T 8af63a4acd194c6a9c496b3318069ac9 P 4024b06066bb48bc9586205d0cdb9f10: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260929 20:57:47.076181 19062 log.cc:824] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:47.079829 19062 tablet_bootstrap.cc:492] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112: No bootstrap required, opened a new log
I20260929 20:57:47.080114 19062 ts_tablet_manager.cc:1397] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 20:57:47.083351 19061 raft_consensus.cc:348] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f857491087b6409b8b607eddc47f6152" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34463 } } peers { permanent_uuid: "04d5f0b685c3461cbd4286c321aa0112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42769 } } peers { permanent_uuid: "4024b06066bb48bc9586205d0cdb9f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32937 } }
I20260929 20:57:47.083724 19061 raft_consensus.cc:374] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:47.083832 19061 raft_consensus.cc:729] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f857491087b6409b8b607eddc47f6152, State: Initialized, Role: FOLLOWER
I20260929 20:57:47.084408 19061 consensus_queue.cc:260] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f857491087b6409b8b607eddc47f6152" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34463 } } peers { permanent_uuid: "04d5f0b685c3461cbd4286c321aa0112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42769 } } peers { permanent_uuid: "4024b06066bb48bc9586205d0cdb9f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32937 } }
I20260929 20:57:47.085449 18910 heartbeater.cc:503] Master 127.0.97.62:44029 was elected leader, sending a full tablet report...
I20260929 20:57:47.085307 19063 raft_consensus.cc:348] T 8af63a4acd194c6a9c496b3318069ac9 P 4024b06066bb48bc9586205d0cdb9f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f857491087b6409b8b607eddc47f6152" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34463 } } peers { permanent_uuid: "04d5f0b685c3461cbd4286c321aa0112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42769 } } peers { permanent_uuid: "4024b06066bb48bc9586205d0cdb9f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32937 } }
I20260929 20:57:47.085716 19063 raft_consensus.cc:374] T 8af63a4acd194c6a9c496b3318069ac9 P 4024b06066bb48bc9586205d0cdb9f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:47.085793 19063 raft_consensus.cc:729] T 8af63a4acd194c6a9c496b3318069ac9 P 4024b06066bb48bc9586205d0cdb9f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4024b06066bb48bc9586205d0cdb9f10, State: Initialized, Role: FOLLOWER
I20260929 20:57:47.086174 19061 ts_tablet_manager.cc:1428] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260929 20:57:47.086441 19063 consensus_queue.cc:260] T 8af63a4acd194c6a9c496b3318069ac9 P 4024b06066bb48bc9586205d0cdb9f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f857491087b6409b8b607eddc47f6152" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34463 } } peers { permanent_uuid: "04d5f0b685c3461cbd4286c321aa0112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42769 } } peers { permanent_uuid: "4024b06066bb48bc9586205d0cdb9f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32937 } }
I20260929 20:57:47.088114 19063 ts_tablet_manager.cc:1428] T 8af63a4acd194c6a9c496b3318069ac9 P 4024b06066bb48bc9586205d0cdb9f10: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260929 20:57:47.089994 19062 raft_consensus.cc:348] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f857491087b6409b8b607eddc47f6152" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34463 } } peers { permanent_uuid: "04d5f0b685c3461cbd4286c321aa0112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42769 } } peers { permanent_uuid: "4024b06066bb48bc9586205d0cdb9f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32937 } }
I20260929 20:57:47.090442 19062 raft_consensus.cc:374] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:47.090543 19062 raft_consensus.cc:729] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04d5f0b685c3461cbd4286c321aa0112, State: Initialized, Role: FOLLOWER
I20260929 20:57:47.091064 19062 consensus_queue.cc:260] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f857491087b6409b8b607eddc47f6152" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34463 } } peers { permanent_uuid: "04d5f0b685c3461cbd4286c321aa0112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42769 } } peers { permanent_uuid: "4024b06066bb48bc9586205d0cdb9f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32937 } }
I20260929 20:57:47.092120 19043 heartbeater.cc:503] Master 127.0.97.62:44029 was elected leader, sending a full tablet report...
I20260929 20:57:47.093777 19062 ts_tablet_manager.cc:1428] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 20:57:47.166581 19069 raft_consensus.cc:482] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:57:47.166875 19069 raft_consensus.cc:504] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f857491087b6409b8b607eddc47f6152" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34463 } } peers { permanent_uuid: "04d5f0b685c3461cbd4286c321aa0112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42769 } } peers { permanent_uuid: "4024b06066bb48bc9586205d0cdb9f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32937 } }
I20260929 20:57:47.168592 19069 leader_election.cc:288] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f857491087b6409b8b607eddc47f6152 (127.0.97.2:34463), 4024b06066bb48bc9586205d0cdb9f10 (127.0.97.1:32937)
I20260929 20:57:47.174037 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.168968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34463 (local address 127.0.97.3:51433)
0929 20:57:47.169184 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:47.169202 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:47.169317 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:47.169842 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:47.169847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:47.169886 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:47.170053 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:47.170062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.171028 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.171033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:47.172291 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:47.172302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.173356 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.173362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:47.173531 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:47.173555 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:57:47.173633 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 20:57:47.173756 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 20:57:47.174042 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.169070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51433 (local address 127.0.97.2:34463)
0929 20:57:47.169292 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:47.169298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:47.169318 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:57:47.169486 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:47.169493 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:47.169567 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:47.169659 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:47.170215 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.170891 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.172447 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.173206 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.173321 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:47.173413 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:47.173677 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 20:57:47.173799 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 20:57:47.174791 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.169580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48771 (local address 127.0.97.1:32937)
0929 20:57:47.169762 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:47.169767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:47.170107 (+ 340us) server_negotiation.cc:407] Connection header received
0929 20:57:47.170283 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:47.170286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:47.170353 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:47.170426 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:47.170898 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.171587 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.173226 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.173979 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.174100 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:47.174223 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:47.174516 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 20:57:47.174616 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:57:47.174896 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.169371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32937 (local address 127.0.97.3:48771)
0929 20:57:47.170013 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:47.170027 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:47.170134 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:47.170455 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:47.170459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:47.170482 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:47.170763 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:47.170771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.171730 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.171734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:47.173078 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:47.173090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.174173 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.174179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:47.174368 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:47.174391 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:57:47.174477 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 20:57:47.174593 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":372,"threads_started":1}
I20260929 20:57:47.175282 18865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af63a4acd194c6a9c496b3318069ac9" candidate_uuid: "04d5f0b685c3461cbd4286c321aa0112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f857491087b6409b8b607eddc47f6152" is_pre_election: true
I20260929 20:57:47.175824 18865 raft_consensus.cc:2454] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04d5f0b685c3461cbd4286c321aa0112 in term 0.
I20260929 20:57:47.175940 18731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af63a4acd194c6a9c496b3318069ac9" candidate_uuid: "04d5f0b685c3461cbd4286c321aa0112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4024b06066bb48bc9586205d0cdb9f10" is_pre_election: true
I20260929 20:57:47.176425 18731 raft_consensus.cc:2454] T 8af63a4acd194c6a9c496b3318069ac9 P 4024b06066bb48bc9586205d0cdb9f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04d5f0b685c3461cbd4286c321aa0112 in term 0.
I20260929 20:57:47.176537 18933 leader_election.cc:302] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04d5f0b685c3461cbd4286c321aa0112, f857491087b6409b8b607eddc47f6152; no voters:
I20260929 20:57:47.176921 19069 raft_consensus.cc:2781] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:57:47.177065 19069 raft_consensus.cc:482] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:57:47.177169 19069 raft_consensus.cc:3037] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:47.179425 19069 raft_consensus.cc:504] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f857491087b6409b8b607eddc47f6152" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34463 } } peers { permanent_uuid: "04d5f0b685c3461cbd4286c321aa0112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42769 } } peers { permanent_uuid: "4024b06066bb48bc9586205d0cdb9f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32937 } }
I20260929 20:57:47.179991 19069 leader_election.cc:288] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [CANDIDATE]: Term 1 election: Requested vote from peers f857491087b6409b8b607eddc47f6152 (127.0.97.2:34463), 4024b06066bb48bc9586205d0cdb9f10 (127.0.97.1:32937)
I20260929 20:57:47.180369 18865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af63a4acd194c6a9c496b3318069ac9" candidate_uuid: "04d5f0b685c3461cbd4286c321aa0112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f857491087b6409b8b607eddc47f6152"
I20260929 20:57:47.180469 18731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af63a4acd194c6a9c496b3318069ac9" candidate_uuid: "04d5f0b685c3461cbd4286c321aa0112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4024b06066bb48bc9586205d0cdb9f10"
I20260929 20:57:47.180546 18865 raft_consensus.cc:3037] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:47.180668 18731 raft_consensus.cc:3037] T 8af63a4acd194c6a9c496b3318069ac9 P 4024b06066bb48bc9586205d0cdb9f10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:47.182796 18865 raft_consensus.cc:2454] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04d5f0b685c3461cbd4286c321aa0112 in term 1.
I20260929 20:57:47.182806 18731 raft_consensus.cc:2454] T 8af63a4acd194c6a9c496b3318069ac9 P 4024b06066bb48bc9586205d0cdb9f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04d5f0b685c3461cbd4286c321aa0112 in term 1.
I20260929 20:57:47.183156 18933 leader_election.cc:302] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04d5f0b685c3461cbd4286c321aa0112, f857491087b6409b8b607eddc47f6152; no voters:
I20260929 20:57:47.183406 19069 raft_consensus.cc:2781] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:57:47.183790 19069 raft_consensus.cc:686] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [term 1 LEADER]: Becoming Leader. State: Replica: 04d5f0b685c3461cbd4286c321aa0112, State: Running, Role: LEADER
I20260929 20:57:47.184298 19069 consensus_queue.cc:237] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f857491087b6409b8b607eddc47f6152" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34463 } } peers { permanent_uuid: "04d5f0b685c3461cbd4286c321aa0112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42769 } } peers { permanent_uuid: "4024b06066bb48bc9586205d0cdb9f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32937 } }
I20260929 20:57:47.188562 18586 catalog_manager.cc:5495] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04d5f0b685c3461cbd4286c321aa0112 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "04d5f0b685c3461cbd4286c321aa0112" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f857491087b6409b8b607eddc47f6152" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04d5f0b685c3461cbd4286c321aa0112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4024b06066bb48bc9586205d0cdb9f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32937 } health_report { overall_health: UNKNOWN } } }
I20260929 20:57:47.205449 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.199994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49434 (local address 127.0.97.3:42769)
0929 20:57:47.200200 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:47.200207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:47.200227 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:57:47.200461 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:47.200466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:47.200568 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:47.200698 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:57:47.201111 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.201839 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.203313 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.203709 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.203829 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:47.204361 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:57:47.204876 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:57:47.205036 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:47.205166 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 20:57:47.205212 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:57:47.213769 18731 raft_consensus.cc:1261] T 8af63a4acd194c6a9c496b3318069ac9 P 4024b06066bb48bc9586205d0cdb9f10 [term 1 FOLLOWER]: Refusing update from remote peer 04d5f0b685c3461cbd4286c321aa0112: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:57:47.213769 18865 raft_consensus.cc:1261] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152 [term 1 FOLLOWER]: Refusing update from remote peer 04d5f0b685c3461cbd4286c321aa0112: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 20:57:47.214932 19044 tablet.cc:2369] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:57:47.214818 19071 consensus_queue.cc:1035] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4024b06066bb48bc9586205d0cdb9f10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:57:47.215292 19069 consensus_queue.cc:1035] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f857491087b6409b8b607eddc47f6152" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:57:47.225615 18911 tablet.cc:2369] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:57:47.229089 19074 mvcc.cc:205] Tried to move back new op lower bound from 7334770553696899072 to 7334770553590640640. Current Snapshot: MvccSnapshot[applied={T|T < 7334770553696899072}]
I20260929 20:57:47.242358 18587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:57:47.242664 18587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:57:47.246284 18587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50de1f83e349487c98eb41909c81302e: Sending DeleteTablet for 3 replicas of tablet 8af63a4acd194c6a9c496b3318069ac9
I20260929 20:57:47.247411 18978 tablet_service.cc:1514] Processing DeleteTablet for tablet 8af63a4acd194c6a9c496b3318069ac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:57:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49426
I20260929 20:57:47.247567 18711 tablet_service.cc:1514] Processing DeleteTablet for tablet 8af63a4acd194c6a9c496b3318069ac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:57:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45648
I20260929 20:57:47.247787 18845 tablet_service.cc:1514] Processing DeleteTablet for tablet 8af63a4acd194c6a9c496b3318069ac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:57:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34320
I20260929 20:57:47.251093 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 18648
I20260929 20:57:47.270067 19084 tablet_replica.cc:330] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112: stopping tablet replica
I20260929 20:57:47.272691 19084 raft_consensus.cc:2229] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [term 1 LEADER]: Raft consensus shutting down.
Sep 29 20:57:47 dist-test-slave-m9g1 krb5kdc[18542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:57:47.277443 19084 raft_consensus.cc:2258] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:57:47.280707 19084 ts_tablet_manager.cc:1905] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:57:47.281563 19085 tablet_replica.cc:330] T 8af63a4acd194c6a9c496b3318069ac9 P 4024b06066bb48bc9586205d0cdb9f10: stopping tablet replica
I20260929 20:57:47.282104 19085 raft_consensus.cc:2229] T 8af63a4acd194c6a9c496b3318069ac9 P 4024b06066bb48bc9586205d0cdb9f10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:57:47.282706 19085 raft_consensus.cc:2258] T 8af63a4acd194c6a9c496b3318069ac9 P 4024b06066bb48bc9586205d0cdb9f10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:57:47.284035 19086 tablet_replica.cc:330] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152: stopping tablet replica
I20260929 20:57:47.284466 19086 raft_consensus.cc:2229] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:57:47.284945 19086 raft_consensus.cc:2258] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:57:47.287681 19085 ts_tablet_manager.cc:1905] T 8af63a4acd194c6a9c496b3318069ac9 P 4024b06066bb48bc9586205d0cdb9f10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:57:47.290267 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.252542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.0.97.1:32937)
0929 20:57:47.252724 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:47.252730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:47.263537 (+ 10807us) server_negotiation.cc:407] Connection header received
0929 20:57:47.264050 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:47.264058 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:47.264159 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:47.264237 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:47.266214 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.267524 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.270394 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.271158 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.271268 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:47.283642 (+ 12374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:47.283661 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:47.283680 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:47.283746 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:47.286608 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:47.287523 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:47.287532 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:47.287535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:47.287708 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:47.288188 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:47.288192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:47.288195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:47.288712 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:47.289024 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:47.289622 (+ 598us) server_negotiation.cc:299] Negotiation successful
0929 20:57:47.289865 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:57:47.291627 18771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:47.292565 19084 ts_tablet_manager.cc:1918] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:57:47.292760 19084 log.cc:1196] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/8af63a4acd194c6a9c496b3318069ac9
I20260929 20:57:47.293327 19084 ts_tablet_manager.cc:1939] T 8af63a4acd194c6a9c496b3318069ac9 P 04d5f0b685c3461cbd4286c321aa0112: Deleting consensus metadata
I20260929 20:57:47.295532 18574 catalog_manager.cc:4841] TS 04d5f0b685c3461cbd4286c321aa0112 (127.0.97.3:42769): tablet 8af63a4acd194c6a9c496b3318069ac9 (table test-table [id=e0652337c09d4d2baaf4141b57fd74e4]) successfully deleted
I20260929 20:57:47.297816 19086 ts_tablet_manager.cc:1905] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:57:47.304894 19086 ts_tablet_manager.cc:1918] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:57:47.305079 19086 log.cc:1196] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/8af63a4acd194c6a9c496b3318069ac9
I20260929 20:57:47.305671 19086 ts_tablet_manager.cc:1939] T 8af63a4acd194c6a9c496b3318069ac9 P f857491087b6409b8b607eddc47f6152: Deleting consensus metadata
I20260929 20:57:47.307669 18573 catalog_manager.cc:4841] TS f857491087b6409b8b607eddc47f6152 (127.0.97.2:34463): tablet 8af63a4acd194c6a9c496b3318069ac9 (table test-table [id=e0652337c09d4d2baaf4141b57fd74e4]) successfully deleted
I20260929 20:57:47.387535 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 18648
W20260929 20:57:47.417089 18573 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:32937 (error 108)
I20260929 20:57:47.417338 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 18782
I20260929 20:57:47.419281 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.418799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32937 (local address 127.0.0.1:45670)
0929 20:57:47.418993 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:47.419111 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 20:57:47.419587 18573 catalog_manager.cc:4584] TS 4024b06066bb48bc9586205d0cdb9f10 (127.0.97.1:32937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8af63a4acd194c6a9c496b3318069ac9: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32937: connect: Connection refused (error 111)
Sep 29 20:57:47 dist-test-slave-m9g1 krb5kdc[18542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:57:47.432842 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.418517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34334 (local address 127.0.97.2:34463)
0929 20:57:47.418684 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:47.418689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:47.418709 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:57:47.418775 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:47.418781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:47.418873 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:47.418986 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:47.420331 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.421097 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.422420 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.422828 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.422941 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:47.427239 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:47.427256 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:47.427273 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:47.427309 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:47.429994 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:47.430824 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:47.430832 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:47.430834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:47.430959 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:47.431329 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:47.431333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:47.431335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:47.431709 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:47.432001 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:47.432309 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 20:57:47.432532 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 20:57:47.433554 18905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:47.473521 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.472959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32937 (local address 127.0.0.1:45686)
0929 20:57:47.473214 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:47.473333 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 20:57:47.474717 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.474310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32937 (local address 127.0.0.1:45700)
0929 20:57:47.474466 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:47.474570 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 20:57:47.521426 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 18782
I20260929 20:57:47.535399 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.534804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32937 (local address 127.0.0.1:45704)
0929 20:57:47.535059 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:47.535176 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 20:57:47.535912 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 18915
I20260929 20:57:47.536806 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.536407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32937 (local address 127.0.0.1:45714)
0929 20:57:47.536563 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:47.536647 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
Sep 29 20:57:47 dist-test-slave-m9g1 krb5kdc[18542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:57:47.554304 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.536863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.0.97.3:42769)
0929 20:57:47.537034 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:47.537040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:47.537061 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:57:47.537210 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:47.537216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:47.537290 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:47.537380 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:47.539102 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.539981 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.541931 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:47.542574 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:47.542677 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:47.548197 (+ 5520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:47.548213 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:47.548233 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:47.548272 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:47.551403 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:47.552278 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:47.552285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:47.552287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:47.552458 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:47.552860 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:47.552866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:47.552868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:47.553237 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:47.553501 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:47.553798 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 20:57:47.553971 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:57:47.555065 19038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:47.617763 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.617227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32937 (local address 127.0.0.1:45730)
0929 20:57:47.617470 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:47.617581 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 20:57:47.618903 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.618443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32937 (local address 127.0.0.1:45742)
0929 20:57:47.618661 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:47.618754 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260929 20:57:47.677639 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 18915
I20260929 20:57:47.693323 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 18555
I20260929 20:57:47.694494 18617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:47.804607 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:47.804063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32937 (local address 127.0.0.1:45756)
0929 20:57:47.804255 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:47.804356 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 20:57:47.805125 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 18555
2026-09-29T20:57:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4103 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T20:57:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:57:47Z Disabled control of system clock
I20260929 20:57:47.874456 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38971
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:57:48.194588 19096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:48.194880 19096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:48.194994 19096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:48.202118 19096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:57:48.202247 19096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:48.202291 19096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:48.202369 19096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:57:48.202422 19096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:57:48.211681 19096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38971
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:48.213006 19096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:48.214811 19096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:48.225832 19104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:48.225914 19102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:48.226048 19101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:48.226511 19096 server_base.cc:1034] running on GCE node
I20260929 20:57:48.227272 19096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:48.228590 19096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:48.229800 19096 hybrid_clock.cc:648] HybridClock initialized: now 1790715468229746 us; error 42 us; skew 500 ppm
I20260929 20:57:48.230140 19096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:57:48.232359 19096 webserver.cc:460] Webserver started at http://127.0.97.62:36339/ using document root <none> and password file <none>
I20260929 20:57:48.233037 19096 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:48.233137 19096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:48.233433 19096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:48.236366 19096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "9b63339ceec946409d017564b95b5e19"
format_stamp: "Formatted at 2026-09-29 20:57:48 on dist-test-slave-m9g1"
server_key: "1a7bea208e6e54d0fed8e3e104553520"
server_key_iv: "4e0f7399299fb540aa241ed1c6a47e9b"
server_key_version: "encryptionkey@0"
I20260929 20:57:48.237164 19096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "9b63339ceec946409d017564b95b5e19"
format_stamp: "Formatted at 2026-09-29 20:57:48 on dist-test-slave-m9g1"
server_key: "1a7bea208e6e54d0fed8e3e104553520"
server_key_iv: "4e0f7399299fb540aa241ed1c6a47e9b"
server_key_version: "encryptionkey@0"
I20260929 20:57:48.242236 19096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:57:48.245641 19110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:48.247004 19096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:57:48.247155 19096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "9b63339ceec946409d017564b95b5e19"
format_stamp: "Formatted at 2026-09-29 20:57:48 on dist-test-slave-m9g1"
server_key: "1a7bea208e6e54d0fed8e3e104553520"
server_key_iv: "4e0f7399299fb540aa241ed1c6a47e9b"
server_key_version: "encryptionkey@0"
I20260929 20:57:48.247346 19096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:48.261148 19096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:48.261924 19096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:48.262179 19096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:48.279091 19096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38971
I20260929 20:57:48.279110 19161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38971 every 8 connection(s)
I20260929 20:57:48.280426 19096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:57:48.284145 19162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:48.287892 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 19096
I20260929 20:57:48.288069 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:57:48.288631 388 external_mini_cluster.cc:1428] Setting key 3051c00aa4447efad4f2c9cb2e7f1f0a
I20260929 20:57:48.296891 19162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19: Bootstrap starting.
I20260929 20:57:48.300000 19162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:48.301083 19162 log.cc:824] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:48.302356 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:48.290754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58682 (local address 127.0.97.62:38971)
0929 20:57:48.291796 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:48.291808 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:48.291849 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:57:48.293476 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:48.293498 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:48.294013 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:48.294447 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:48.295051 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:48.296288 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:48.297642 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:48.298443 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:48.298653 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:48.299752 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:48.299759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:48.299774 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:48.300351 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:48.300455 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:48.300481 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:48.300582 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:48.300831 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 20:57:48.301128 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":379,"threads_started":1}
I20260929 20:57:48.304205 19162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19: No bootstrap required, opened a new log
I20260929 20:57:48.311581 19162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b63339ceec946409d017564b95b5e19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38971 } }
I20260929 20:57:48.311981 19162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:48.312077 19162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b63339ceec946409d017564b95b5e19, State: Initialized, Role: FOLLOWER
I20260929 20:57:48.312631 19162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b63339ceec946409d017564b95b5e19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38971 } }
I20260929 20:57:48.312851 19162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:57:48.312974 19162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:57:48.313118 19162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:48.315471 19162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b63339ceec946409d017564b95b5e19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38971 } }
I20260929 20:57:48.315987 19162 leader_election.cc:302] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b63339ceec946409d017564b95b5e19; no voters:
I20260929 20:57:48.316526 19162 leader_election.cc:288] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:57:48.316665 19167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:57:48.317322 19167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19 [term 1 LEADER]: Becoming Leader. State: Replica: 9b63339ceec946409d017564b95b5e19, State: Running, Role: LEADER
I20260929 20:57:48.317827 19167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b63339ceec946409d017564b95b5e19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38971 } }
I20260929 20:57:48.318509 19162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:57:48.320451 19168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b63339ceec946409d017564b95b5e19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b63339ceec946409d017564b95b5e19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38971 } } }
I20260929 20:57:48.321099 19169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b63339ceec946409d017564b95b5e19. Latest consensus state: current_term: 1 leader_uuid: "9b63339ceec946409d017564b95b5e19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b63339ceec946409d017564b95b5e19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38971 } } }
I20260929 20:57:48.321406 19168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19 [sys.catalog]: This master's current role is: LEADER
I20260929 20:57:48.321635 19169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19 [sys.catalog]: This master's current role is: LEADER
I20260929 20:57:48.323524 19178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:57:48.327739 19178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:57:48.336144 19178 catalog_manager.cc:1347] Generated new cluster ID: b6a10bfe854a44cc81890b059a4ce08b
I20260929 20:57:48.336251 19178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:57:48.355329 19178 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:57:48.356634 19178 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:57:48.368976 19178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19: Generated new TSK 0
I20260929 20:57:48.369851 19178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:57:48.385142 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38971
--builtin_ntp_servers=127.0.97.20:43309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:57:48.713868 19186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:48.714175 19186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:48.714319 19186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:48.721762 19186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:48.721880 19186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:48.722021 19186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:57:48.731614 19186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:48.732951 19186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:48.734838 19186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:48.748412 19191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:48.748389 19192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:48.748456 19194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:48.749276 19186 server_base.cc:1034] running on GCE node
I20260929 20:57:48.749855 19186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:48.750852 19186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:48.752086 19186 hybrid_clock.cc:648] HybridClock initialized: now 1790715468752036 us; error 42 us; skew 500 ppm
I20260929 20:57:48.752409 19186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:57:48.754663 19186 webserver.cc:460] Webserver started at http://127.0.97.1:44381/ using document root <none> and password file <none>
I20260929 20:57:48.755369 19186 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:48.755465 19186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:48.755776 19186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:48.758658 19186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "2fc1fbfc91964c7eaa3b3c2c8cf09a5e"
format_stamp: "Formatted at 2026-09-29 20:57:48 on dist-test-slave-m9g1"
server_key: "2d702161c116eebf8150ad13c6d9a76e"
server_key_iv: "b1335852f03523bc78ee125ff6163147"
server_key_version: "encryptionkey@0"
I20260929 20:57:48.759449 19186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2fc1fbfc91964c7eaa3b3c2c8cf09a5e"
format_stamp: "Formatted at 2026-09-29 20:57:48 on dist-test-slave-m9g1"
server_key: "2d702161c116eebf8150ad13c6d9a76e"
server_key_iv: "b1335852f03523bc78ee125ff6163147"
server_key_version: "encryptionkey@0"
I20260929 20:57:48.764577 19186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:57:48.768224 19200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:48.769649 19186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:57:48.769804 19186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "2fc1fbfc91964c7eaa3b3c2c8cf09a5e"
format_stamp: "Formatted at 2026-09-29 20:57:48 on dist-test-slave-m9g1"
server_key: "2d702161c116eebf8150ad13c6d9a76e"
server_key_iv: "b1335852f03523bc78ee125ff6163147"
server_key_version: "encryptionkey@0"
I20260929 20:57:48.770022 19186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:48.799947 19186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:48.800812 19186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:48.801086 19186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:48.802039 19186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:48.803646 19186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:48.803748 19186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:48.803858 19186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:48.803931 19186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:48.834255 19186 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39569
I20260929 20:57:48.834303 19312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39569 every 8 connection(s)
I20260929 20:57:48.835537 19186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:57:48.838423 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 19186
I20260929 20:57:48.838605 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:57:48.839133 388 external_mini_cluster.cc:1428] Setting key 075a0b4beb3cc495ab7a8739ecf38d44
I20260929 20:57:48.842006 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38971
--builtin_ntp_servers=127.0.97.20:43309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:57:48.849268 19315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:48.838134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54067 (local address 127.0.97.62:38971)
0929 20:57:48.838673 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:48.838678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:48.840413 (+ 1735us) server_negotiation.cc:407] Connection header received
0929 20:57:48.842634 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:48.842640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:48.842731 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:48.842827 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:48.844090 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:48.844843 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:48.846501 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:48.846897 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:48.847008 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:48.848357 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:48.848361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:48.848364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:48.848385 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:48.848465 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:48.848480 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:48.848595 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:48.848799 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 20:57:48.848902 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":292,"threads_started":1}
I20260929 20:57:48.850386 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:48.838986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38971 (local address 127.0.97.1:54067)
0929 20:57:48.840256 (+ 1270us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:48.840298 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:48.842345 (+ 2047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:48.843139 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:48.843149 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:48.843254 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:57:48.843882 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:48.843929 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:48.845001 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:48.845005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:48.846355 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:48.846369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:48.847123 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:48.847130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:48.847360 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:48.848032 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:57:48.848035 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:57:48.848084 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:57:48.848167 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:57:48.848581 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:57:48.848594 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:57:48.848603 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 20:57:48.848776 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 20:57:48.849204 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":348,"threads_started":1}
I20260929 20:57:48.851990 19313 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38971
I20260929 20:57:48.852345 19313 heartbeater.cc:464] Registering TS with master...
I20260929 20:57:48.853241 19313 heartbeater.cc:511] Master 127.0.97.62:38971 requested a full tablet report, sending...
I20260929 20:57:48.855501 19127 ts_manager.cc:194] Registered new tserver with Master: 2fc1fbfc91964c7eaa3b3c2c8cf09a5e (127.0.97.1:39569)
I20260929 20:57:48.857051 19127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:54067
W20260929 20:57:49.168382 19317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:49.168665 19317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:49.168776 19317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:49.176262 19317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:49.176370 19317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:49.176507 19317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:57:49.186030 19317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:49.187328 19317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:49.189111 19317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:49.201978 19322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:49.202009 19323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:49.201949 19325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:49.202677 19317 server_base.cc:1034] running on GCE node
I20260929 20:57:49.203243 19317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:49.204183 19317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:49.205379 19317 hybrid_clock.cc:648] HybridClock initialized: now 1790715469205317 us; error 58 us; skew 500 ppm
I20260929 20:57:49.205687 19317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:57:49.207867 19317 webserver.cc:460] Webserver started at http://127.0.97.2:33299/ using document root <none> and password file <none>
I20260929 20:57:49.208551 19317 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:49.208648 19317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:49.208935 19317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:49.211869 19317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "2027f84aa41a49c38fb6c3a900d1573e"
format_stamp: "Formatted at 2026-09-29 20:57:49 on dist-test-slave-m9g1"
server_key: "5cd21c02695b702ac77f888f7309c0a6"
server_key_iv: "f40c55ed765b26ae0d6ab21c7eefa124"
server_key_version: "encryptionkey@0"
I20260929 20:57:49.212651 19317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2027f84aa41a49c38fb6c3a900d1573e"
format_stamp: "Formatted at 2026-09-29 20:57:49 on dist-test-slave-m9g1"
server_key: "5cd21c02695b702ac77f888f7309c0a6"
server_key_iv: "f40c55ed765b26ae0d6ab21c7eefa124"
server_key_version: "encryptionkey@0"
I20260929 20:57:49.217757 19317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:57:49.221313 19331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:49.222868 19317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:57:49.223032 19317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "2027f84aa41a49c38fb6c3a900d1573e"
format_stamp: "Formatted at 2026-09-29 20:57:49 on dist-test-slave-m9g1"
server_key: "5cd21c02695b702ac77f888f7309c0a6"
server_key_iv: "f40c55ed765b26ae0d6ab21c7eefa124"
server_key_version: "encryptionkey@0"
I20260929 20:57:49.223240 19317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:49.236315 19317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:49.237112 19317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:49.237371 19317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:49.238314 19317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:49.239766 19317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:49.239866 19317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:49.239975 19317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:49.240047 19317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:49.269470 19317 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45655
I20260929 20:57:49.269511 19443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45655 every 8 connection(s)
I20260929 20:57:49.270788 19317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:57:49.276234 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 19317
I20260929 20:57:49.276372 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:57:49.276787 388 external_mini_cluster.cc:1428] Setting key 76f8362843715a00ed55a2a55923ea8c
I20260929 20:57:49.279743 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38971
--builtin_ntp_servers=127.0.97.20:43309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:57:49.284842 19315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:49.273288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54301 (local address 127.0.97.62:38971)
0929 20:57:49.273489 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:49.273493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:49.275140 (+ 1647us) server_negotiation.cc:407] Connection header received
0929 20:57:49.277015 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:49.277021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:49.277110 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:49.277199 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:49.278356 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.279393 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.281717 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.282288 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.282410 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:49.284010 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:49.284017 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:49.284021 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:49.284048 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:49.284092 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:49.284107 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:49.284194 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:49.284418 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 20:57:49.284527 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:57:49.286115 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:49.273949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38971 (local address 127.0.97.2:54301)
0929 20:57:49.274968 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:49.275005 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:49.276729 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:49.277452 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:49.277461 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:49.277554 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:57:49.278135 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:49.278155 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.279560 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.279565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:49.281553 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:49.281569 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.282516 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.282523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:49.282822 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:49.283612 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:57:49.283616 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:57:49.283674 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:57:49.283788 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:57:49.284206 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:57:49.284217 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:57:49.284226 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 20:57:49.284397 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 20:57:49.284815 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":353,"threads_started":1}
I20260929 20:57:49.287705 19444 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38971
I20260929 20:57:49.288053 19444 heartbeater.cc:464] Registering TS with master...
I20260929 20:57:49.288936 19444 heartbeater.cc:511] Master 127.0.97.62:38971 requested a full tablet report, sending...
I20260929 20:57:49.290490 19127 ts_manager.cc:194] Registered new tserver with Master: 2027f84aa41a49c38fb6c3a900d1573e (127.0.97.2:45655)
I20260929 20:57:49.291589 19127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:54301
W20260929 20:57:49.603235 19447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:49.603570 19447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:49.603706 19447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:49.610954 19447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:49.611078 19447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:49.611230 19447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:57:49.620389 19447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:49.621726 19447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:49.623589 19447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:49.636121 19452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:49.636081 19453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:49.636162 19455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:49.636991 19447 server_base.cc:1034] running on GCE node
I20260929 20:57:49.637573 19447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:49.638559 19447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:49.639755 19447 hybrid_clock.cc:648] HybridClock initialized: now 1790715469639704 us; error 48 us; skew 500 ppm
I20260929 20:57:49.640061 19447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:57:49.642365 19447 webserver.cc:460] Webserver started at http://127.0.97.3:34367/ using document root <none> and password file <none>
I20260929 20:57:49.643034 19447 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:49.643131 19447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:49.643416 19447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:49.646284 19447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "6b64bf7fc2c74853879c7a809b00e73b"
format_stamp: "Formatted at 2026-09-29 20:57:49 on dist-test-slave-m9g1"
server_key: "0c069c38b4426af36dd994f1f5496559"
server_key_iv: "b8ee992dfe174e28227ae5bb8a50b97c"
server_key_version: "encryptionkey@0"
I20260929 20:57:49.647115 19447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6b64bf7fc2c74853879c7a809b00e73b"
format_stamp: "Formatted at 2026-09-29 20:57:49 on dist-test-slave-m9g1"
server_key: "0c069c38b4426af36dd994f1f5496559"
server_key_iv: "b8ee992dfe174e28227ae5bb8a50b97c"
server_key_version: "encryptionkey@0"
I20260929 20:57:49.652087 19447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260929 20:57:49.655339 19461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:49.656682 19447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:57:49.656842 19447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "6b64bf7fc2c74853879c7a809b00e73b"
format_stamp: "Formatted at 2026-09-29 20:57:49 on dist-test-slave-m9g1"
server_key: "0c069c38b4426af36dd994f1f5496559"
server_key_iv: "b8ee992dfe174e28227ae5bb8a50b97c"
server_key_version: "encryptionkey@0"
I20260929 20:57:49.657043 19447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:49.671440 19447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:49.672267 19447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:49.672515 19447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:49.673430 19447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:49.674988 19447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:49.675079 19447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:49.675187 19447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:49.675253 19447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:49.704694 19447 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33231
I20260929 20:57:49.704742 19573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33231 every 8 connection(s)
I20260929 20:57:49.705991 19447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:57:49.715428 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 19447
I20260929 20:57:49.715601 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:57:49.716111 388 external_mini_cluster.cc:1428] Setting key 262cb6129e6840d947f3bedbdf634f73
I20260929 20:57:49.718070 19315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:49.708391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60517 (local address 127.0.97.62:38971)
0929 20:57:49.708643 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:49.708650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:49.710040 (+ 1390us) server_negotiation.cc:407] Connection header received
0929 20:57:49.711824 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:49.711829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:49.711904 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:49.711993 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:49.712995 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.713729 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.715280 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.715884 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.716011 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:49.717294 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:49.717299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:49.717302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:49.717321 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:49.717365 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:49.717377 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:49.717447 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:49.717654 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 20:57:49.717751 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260929 20:57:49.719149 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:49.708993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38971 (local address 127.0.97.3:60517)
0929 20:57:49.709859 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:49.709894 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:49.711568 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:49.712201 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:49.712210 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:49.712300 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:57:49.712848 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:49.712863 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.713848 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.713852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:49.715136 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:49.715148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.716087 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.716092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:49.716303 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:49.716990 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:57:49.716992 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:57:49.717032 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:57:49.717107 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:57:49.717465 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:57:49.717477 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:57:49.717483 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 20:57:49.717633 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 20:57:49.718018 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":274,"threads_started":1}
I20260929 20:57:49.720435 19574 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38971
I20260929 20:57:49.720772 19574 heartbeater.cc:464] Registering TS with master...
I20260929 20:57:49.721529 19574 heartbeater.cc:511] Master 127.0.97.62:38971 requested a full tablet report, sending...
I20260929 20:57:49.722916 19127 ts_manager.cc:194] Registered new tserver with Master: 6b64bf7fc2c74853879c7a809b00e73b (127.0.97.3:33231)
I20260929 20:57:49.723726 19127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:60517
I20260929 20:57:49.731178 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:57:49.740908 19315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:49.735355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33084 (local address 127.0.97.62:38971)
0929 20:57:49.735528 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:49.735532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:49.735935 (+ 403us) server_negotiation.cc:407] Connection header received
0929 20:57:49.736071 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:49.736075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:49.736146 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:49.736230 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:49.736669 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.737341 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.738738 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.739169 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.739269 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:49.740323 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:49.740340 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:49.740344 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:49.740361 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:49.740393 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:49.740405 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:49.740512 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:49.740635 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 20:57:49.740680 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:57:49.744853 19127 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:57:49.751686 19127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:57:49.784448 19588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:49.763818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45655 (local address 127.0.0.1:41330)
0929 20:57:49.775091 (+ 11273us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:49.775148 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:49.775305 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:49.776989 (+ 1684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:49.776995 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:49.777025 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:49.777412 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:49.777440 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.779328 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.779333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:49.781532 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:49.781547 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.783511 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.783517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:49.783716 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:49.783772 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 20:57:49.783877 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 20:57:49.784078 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9871,"spinlock_wait_cycles":9088,"thread_start_us":10982,"threads_started":1}
I20260929 20:57:49.785583 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:49.764067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33231 (local address 127.0.0.1:41156)
0929 20:57:49.775098 (+ 11031us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:49.775145 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:49.775303 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:49.778361 (+ 3058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:49.778366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:49.778396 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:49.778854 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:49.778864 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.780758 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.780763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:49.782858 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:49.782872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.784757 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.784764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:49.784957 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:49.784994 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 20:57:49.785065 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 20:57:49.785196 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10886,"spinlock_wait_cycles":4864,"thread_start_us":14498,"threads_started":1}
I20260929 20:57:49.785733 19587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:49.764328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39569 (local address 127.0.0.1:44984)
0929 20:57:49.778860 (+ 14532us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:49.778879 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:49.779011 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:49.779613 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:49.779622 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:49.779646 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:49.780128 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:49.780141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.781628 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.781633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:49.783428 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:49.783441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.785021 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.785027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:49.785201 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:49.785230 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 20:57:49.785303 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 20:57:49.785427 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14397,"mutex_wait_us":62,"thread_start_us":9202,"threads_started":1}
I20260929 20:57:49.785831 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:49.764671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41330 (local address 127.0.97.2:45655)
0929 20:57:49.776204 (+ 11533us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:49.776224 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:49.776261 (+ 37us) server_negotiation.cc:407] Connection header received
0929 20:57:49.776379 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:49.776390 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:49.776721 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:49.776967 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:49.777644 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.779202 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.781777 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.783290 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.785045 (+ 1755us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:49.785290 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:49.785356 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 20:57:49.785503 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11359,"thread_start_us":11234,"threads_started":1}
I20260929 20:57:49.785933 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:49.765112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41156 (local address 127.0.97.3:33231)
0929 20:57:49.777674 (+ 12562us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:49.777692 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:49.777727 (+ 35us) server_negotiation.cc:407] Connection header received
0929 20:57:49.777827 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:49.777835 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:49.778117 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:49.779047 (+ 930us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:49.779311 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.780618 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.783091 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.784467 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.784787 (+ 320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:49.785078 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:49.785153 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 20:57:49.785316 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12404,"thread_start_us":12258,"threads_started":1}
I20260929 20:57:49.786471 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:49.764993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.0.97.1:39569)
0929 20:57:49.774395 (+ 9402us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:49.774405 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:49.779017 (+ 4612us) server_negotiation.cc:407] Connection header received
0929 20:57:49.779132 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:49.779140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:49.779394 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:49.779628 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:49.780315 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.781491 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.783602 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.784500 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.784618 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:49.784880 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:49.785954 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0929 20:57:49.786109 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9206,"thread_start_us":9446,"threads_started":1}
I20260929 20:57:49.789729 19379 tablet_service.cc:1467] Processing CreateTablet for tablet 4e790a3057ec49d1af1384cb7822c51a (DEFAULT_TABLE table=test-table [id=a648c184fbae4500a0457c6d5cda0af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:57:49.789733 19248 tablet_service.cc:1467] Processing CreateTablet for tablet 4e790a3057ec49d1af1384cb7822c51a (DEFAULT_TABLE table=test-table [id=a648c184fbae4500a0457c6d5cda0af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:57:49.790350 19509 tablet_service.cc:1467] Processing CreateTablet for tablet 4e790a3057ec49d1af1384cb7822c51a (DEFAULT_TABLE table=test-table [id=a648c184fbae4500a0457c6d5cda0af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:57:49.791178 19248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e790a3057ec49d1af1384cb7822c51a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:49.791179 19379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e790a3057ec49d1af1384cb7822c51a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:49.792312 19509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e790a3057ec49d1af1384cb7822c51a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:49.802727 19592 tablet_bootstrap.cc:492] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e: Bootstrap starting.
I20260929 20:57:49.806329 19592 tablet_bootstrap.cc:654] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:49.807305 19592 log.cc:824] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:49.810027 19592 tablet_bootstrap.cc:492] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e: No bootstrap required, opened a new log
I20260929 20:57:49.810322 19592 ts_tablet_manager.cc:1397] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260929 20:57:49.810926 19594 tablet_bootstrap.cc:492] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b: Bootstrap starting.
I20260929 20:57:49.811404 19593 tablet_bootstrap.cc:492] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e: Bootstrap starting.
I20260929 20:57:49.813984 19593 tablet_bootstrap.cc:654] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:49.814365 19594 tablet_bootstrap.cc:654] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:49.815011 19593 log.cc:824] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:49.815793 19594 log.cc:824] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:49.818061 19593 tablet_bootstrap.cc:492] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e: No bootstrap required, opened a new log
I20260929 20:57:49.818423 19593 ts_tablet_manager.cc:1397] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 20:57:49.819689 19594 tablet_bootstrap.cc:492] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b: No bootstrap required, opened a new log
I20260929 20:57:49.820006 19594 ts_tablet_manager.cc:1397] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.006s
I20260929 20:57:49.820245 19592 raft_consensus.cc:348] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2027f84aa41a49c38fb6c3a900d1573e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45655 } } peers { permanent_uuid: "6b64bf7fc2c74853879c7a809b00e73b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33231 } } peers { permanent_uuid: "2fc1fbfc91964c7eaa3b3c2c8cf09a5e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39569 } }
I20260929 20:57:49.820711 19592 raft_consensus.cc:374] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:49.820878 19592 raft_consensus.cc:729] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc1fbfc91964c7eaa3b3c2c8cf09a5e, State: Initialized, Role: FOLLOWER
I20260929 20:57:49.821537 19592 consensus_queue.cc:260] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2027f84aa41a49c38fb6c3a900d1573e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45655 } } peers { permanent_uuid: "6b64bf7fc2c74853879c7a809b00e73b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33231 } } peers { permanent_uuid: "2fc1fbfc91964c7eaa3b3c2c8cf09a5e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39569 } }
I20260929 20:57:49.822700 19313 heartbeater.cc:503] Master 127.0.97.62:38971 was elected leader, sending a full tablet report...
I20260929 20:57:49.823333 19592 ts_tablet_manager.cc:1428] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260929 20:57:49.827879 19593 raft_consensus.cc:348] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2027f84aa41a49c38fb6c3a900d1573e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45655 } } peers { permanent_uuid: "6b64bf7fc2c74853879c7a809b00e73b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33231 } } peers { permanent_uuid: "2fc1fbfc91964c7eaa3b3c2c8cf09a5e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39569 } }
I20260929 20:57:49.828353 19593 raft_consensus.cc:374] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:49.828471 19593 raft_consensus.cc:729] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2027f84aa41a49c38fb6c3a900d1573e, State: Initialized, Role: FOLLOWER
I20260929 20:57:49.828413 19594 raft_consensus.cc:348] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2027f84aa41a49c38fb6c3a900d1573e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45655 } } peers { permanent_uuid: "6b64bf7fc2c74853879c7a809b00e73b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33231 } } peers { permanent_uuid: "2fc1fbfc91964c7eaa3b3c2c8cf09a5e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39569 } }
I20260929 20:57:49.828806 19594 raft_consensus.cc:374] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:49.828895 19594 raft_consensus.cc:729] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b64bf7fc2c74853879c7a809b00e73b, State: Initialized, Role: FOLLOWER
I20260929 20:57:49.829102 19593 consensus_queue.cc:260] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2027f84aa41a49c38fb6c3a900d1573e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45655 } } peers { permanent_uuid: "6b64bf7fc2c74853879c7a809b00e73b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33231 } } peers { permanent_uuid: "2fc1fbfc91964c7eaa3b3c2c8cf09a5e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39569 } }
I20260929 20:57:49.829430 19594 consensus_queue.cc:260] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2027f84aa41a49c38fb6c3a900d1573e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45655 } } peers { permanent_uuid: "6b64bf7fc2c74853879c7a809b00e73b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33231 } } peers { permanent_uuid: "2fc1fbfc91964c7eaa3b3c2c8cf09a5e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39569 } }
I20260929 20:57:49.830279 19444 heartbeater.cc:503] Master 127.0.97.62:38971 was elected leader, sending a full tablet report...
I20260929 20:57:49.830986 19593 ts_tablet_manager.cc:1428] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260929 20:57:49.831424 19574 heartbeater.cc:503] Master 127.0.97.62:38971 was elected leader, sending a full tablet report...
I20260929 20:57:49.833560 19594 ts_tablet_manager.cc:1428] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b: Time spent starting tablet: real 0.013s user 0.010s sys 0.000s
W20260929 20:57:49.837653 19314 tablet.cc:2369] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:57:49.909214 19598 raft_consensus.cc:482] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:57:49.909550 19598 raft_consensus.cc:504] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2027f84aa41a49c38fb6c3a900d1573e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45655 } } peers { permanent_uuid: "6b64bf7fc2c74853879c7a809b00e73b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33231 } } peers { permanent_uuid: "2fc1fbfc91964c7eaa3b3c2c8cf09a5e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39569 } }
I20260929 20:57:49.911413 19598 leader_election.cc:288] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2027f84aa41a49c38fb6c3a900d1573e (127.0.97.2:45655), 6b64bf7fc2c74853879c7a809b00e73b (127.0.97.3:33231)
I20260929 20:57:49.918324 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:49.911913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48873 (local address 127.0.97.3:33231)
0929 20:57:49.912114 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:49.912120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:49.912948 (+ 828us) server_negotiation.cc:407] Connection header received
0929 20:57:49.913090 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:49.913094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:49.913153 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:49.913228 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:49.913858 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.914613 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.916629 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.917423 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.917553 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:49.917646 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:49.917922 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 20:57:49.918049 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 20:57:49.918310 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:49.911914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56907 (local address 127.0.97.2:45655)
0929 20:57:49.912109 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:49.912115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:49.912846 (+ 731us) server_negotiation.cc:407] Connection header received
0929 20:57:49.913080 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:49.913085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:49.913149 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:49.913224 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:49.913814 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.914561 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.916601 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.917368 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.917463 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:49.917554 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:49.917913 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 20:57:49.918038 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:57:49.918463 19601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:49.911844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45655 (local address 127.0.97.1:56907)
0929 20:57:49.912732 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:49.912749 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:49.912905 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:49.913257 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:49.913261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:49.913284 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:49.913677 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:49.913687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.914681 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.914686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:49.916408 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:49.916421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.917551 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.917557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:49.917715 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:49.917777 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 20:57:49.917859 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 20:57:49.917997 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":136,"spinlock_wait_cycles":35200,"thread_start_us":454,"threads_started":1}
I20260929 20:57:49.918529 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:49.911903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33231 (local address 127.0.97.1:48873)
0929 20:57:49.912845 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:49.912864 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:49.912970 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:49.913262 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:49.913266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:49.913359 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:49.913711 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:49.913720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.914739 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.914743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:49.916423 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:49.916434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.917619 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.917623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:49.917752 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:49.917795 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 20:57:49.917881 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 20:57:49.918006 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":153,"spinlock_wait_cycles":25856,"thread_start_us":493,"threads_started":1}
I20260929 20:57:49.919633 19399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e790a3057ec49d1af1384cb7822c51a" candidate_uuid: "2fc1fbfc91964c7eaa3b3c2c8cf09a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2027f84aa41a49c38fb6c3a900d1573e" is_pre_election: true
I20260929 20:57:49.919639 19529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e790a3057ec49d1af1384cb7822c51a" candidate_uuid: "2fc1fbfc91964c7eaa3b3c2c8cf09a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b64bf7fc2c74853879c7a809b00e73b" is_pre_election: true
I20260929 20:57:49.920164 19399 raft_consensus.cc:2454] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc1fbfc91964c7eaa3b3c2c8cf09a5e in term 0.
I20260929 20:57:49.920168 19529 raft_consensus.cc:2454] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc1fbfc91964c7eaa3b3c2c8cf09a5e in term 0.
I20260929 20:57:49.920830 19201 leader_election.cc:302] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc1fbfc91964c7eaa3b3c2c8cf09a5e, 6b64bf7fc2c74853879c7a809b00e73b; no voters:
I20260929 20:57:49.921274 19598 raft_consensus.cc:2781] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:57:49.921439 19598 raft_consensus.cc:482] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:57:49.921555 19598 raft_consensus.cc:3037] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:49.924057 19598 raft_consensus.cc:504] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2027f84aa41a49c38fb6c3a900d1573e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45655 } } peers { permanent_uuid: "6b64bf7fc2c74853879c7a809b00e73b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33231 } } peers { permanent_uuid: "2fc1fbfc91964c7eaa3b3c2c8cf09a5e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39569 } }
I20260929 20:57:49.924778 19598 leader_election.cc:288] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [CANDIDATE]: Term 1 election: Requested vote from peers 2027f84aa41a49c38fb6c3a900d1573e (127.0.97.2:45655), 6b64bf7fc2c74853879c7a809b00e73b (127.0.97.3:33231)
I20260929 20:57:49.925211 19399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e790a3057ec49d1af1384cb7822c51a" candidate_uuid: "2fc1fbfc91964c7eaa3b3c2c8cf09a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2027f84aa41a49c38fb6c3a900d1573e"
I20260929 20:57:49.925357 19529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e790a3057ec49d1af1384cb7822c51a" candidate_uuid: "2fc1fbfc91964c7eaa3b3c2c8cf09a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b64bf7fc2c74853879c7a809b00e73b"
I20260929 20:57:49.925450 19399 raft_consensus.cc:3037] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:49.925570 19529 raft_consensus.cc:3037] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:49.928746 19529 raft_consensus.cc:2454] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc1fbfc91964c7eaa3b3c2c8cf09a5e in term 1.
I20260929 20:57:49.928746 19399 raft_consensus.cc:2454] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc1fbfc91964c7eaa3b3c2c8cf09a5e in term 1.
I20260929 20:57:49.929183 19201 leader_election.cc:302] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc1fbfc91964c7eaa3b3c2c8cf09a5e, 6b64bf7fc2c74853879c7a809b00e73b; no voters:
I20260929 20:57:49.929575 19598 raft_consensus.cc:2781] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:57:49.929997 19598 raft_consensus.cc:686] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [term 1 LEADER]: Becoming Leader. State: Replica: 2fc1fbfc91964c7eaa3b3c2c8cf09a5e, State: Running, Role: LEADER
I20260929 20:57:49.930487 19598 consensus_queue.cc:237] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2027f84aa41a49c38fb6c3a900d1573e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45655 } } peers { permanent_uuid: "6b64bf7fc2c74853879c7a809b00e73b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33231 } } peers { permanent_uuid: "2fc1fbfc91964c7eaa3b3c2c8cf09a5e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39569 } }
I20260929 20:57:49.935320 19126 catalog_manager.cc:5495] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fc1fbfc91964c7eaa3b3c2c8cf09a5e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "2fc1fbfc91964c7eaa3b3c2c8cf09a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2027f84aa41a49c38fb6c3a900d1573e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b64bf7fc2c74853879c7a809b00e73b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fc1fbfc91964c7eaa3b3c2c8cf09a5e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39569 } health_report { overall_health: HEALTHY } } }
W20260929 20:57:49.957617 19575 tablet.cc:2369] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:57:49.975132 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:49.968109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.0.97.1:39569)
0929 20:57:49.968294 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:49.968301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:49.968325 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:57:49.968572 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:49.968578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:49.968717 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:49.968890 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:57:49.969268 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.970327 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.972302 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:49.972719 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:49.972827 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:49.973574 (+ 747us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:57:49.974377 (+ 803us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:57:49.974559 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:49.974757 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 20:57:49.974811 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 20:57:49.984404 19529 raft_consensus.cc:1261] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b [term 1 FOLLOWER]: Refusing update from remote peer 2fc1fbfc91964c7eaa3b3c2c8cf09a5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:57:49.984404 19399 raft_consensus.cc:1261] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e [term 1 FOLLOWER]: Refusing update from remote peer 2fc1fbfc91964c7eaa3b3c2c8cf09a5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:57:49.985481 19603 consensus_queue.cc:1035] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2027f84aa41a49c38fb6c3a900d1573e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:57:49.986336 19598 consensus_queue.cc:1035] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b64bf7fc2c74853879c7a809b00e73b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:57:50.005884 19606 mvcc.cc:205] Tried to move back new op lower bound from 7334770565045149696 to 7334770564839022592. Current Snapshot: MvccSnapshot[applied={T|T < 7334770565045149696}]
I20260929 20:57:50.008198 19608 mvcc.cc:205] Tried to move back new op lower bound from 7334770565045149696 to 7334770564839022592. Current Snapshot: MvccSnapshot[applied={T|T < 7334770565045149696}]
I20260929 20:57:50.019199 19127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:57:50.019508 19127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:57:50.023419 19127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b63339ceec946409d017564b95b5e19: Sending DeleteTablet for 3 replicas of tablet 4e790a3057ec49d1af1384cb7822c51a
I20260929 20:57:50.024840 19248 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e790a3057ec49d1af1384cb7822c51a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:57:50 UTC) from {username='slave'} at 127.0.0.1:44984
I20260929 20:57:50.024840 19379 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e790a3057ec49d1af1384cb7822c51a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:57:50 UTC) from {username='slave'} at 127.0.0.1:41330
I20260929 20:57:50.024837 19509 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e790a3057ec49d1af1384cb7822c51a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:57:50 UTC) from {username='slave'} at 127.0.0.1:41156
I20260929 20:57:50.026347 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 19186
I20260929 20:57:50.047683 19615 tablet_replica.cc:330] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e: stopping tablet replica
I20260929 20:57:50.049153 19615 raft_consensus.cc:2229] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:57:50.051833 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:50.028233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44996 (local address 127.0.97.1:39569)
0929 20:57:50.028442 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:50.028449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:50.039405 (+ 10956us) server_negotiation.cc:407] Connection header received
0929 20:57:50.040008 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:50.040015 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:50.040150 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:50.040231 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:50.040982 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:50.042502 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:50.044821 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:50.045597 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:50.045714 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:50.049619 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:50.049634 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:50.049658 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:50.050641 (+ 983us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:50.050730 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:50.050763 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:50.050885 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:50.051216 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 20:57:50.051358 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:57:50.053494 19308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:50.054394 19616 tablet_replica.cc:330] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e: stopping tablet replica
I20260929 20:57:50.054976 19617 tablet_replica.cc:330] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b: stopping tablet replica
I20260929 20:57:50.055356 19617 raft_consensus.cc:2229] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:57:50.055814 19617 raft_consensus.cc:2258] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:57:50.056494 19616 raft_consensus.cc:2229] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:57:50.056910 19616 raft_consensus.cc:2258] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:57:50.058876 19617 ts_tablet_manager.cc:1905] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:57:50.058923 19616 ts_tablet_manager.cc:1905] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:57:50.066826 19617 ts_tablet_manager.cc:1918] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:57:50.066825 19616 ts_tablet_manager.cc:1918] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:57:50.067021 19616 log.cc:1196] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/4e790a3057ec49d1af1384cb7822c51a
I20260929 20:57:50.067070 19617 log.cc:1196] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/4e790a3057ec49d1af1384cb7822c51a
I20260929 20:57:50.067588 19617 ts_tablet_manager.cc:1939] T 4e790a3057ec49d1af1384cb7822c51a P 6b64bf7fc2c74853879c7a809b00e73b: Deleting consensus metadata
I20260929 20:57:50.067577 19616 ts_tablet_manager.cc:1939] T 4e790a3057ec49d1af1384cb7822c51a P 2027f84aa41a49c38fb6c3a900d1573e: Deleting consensus metadata
I20260929 20:57:50.069551 19111 catalog_manager.cc:4841] TS 6b64bf7fc2c74853879c7a809b00e73b (127.0.97.3:33231): tablet 4e790a3057ec49d1af1384cb7822c51a (table test-table [id=a648c184fbae4500a0457c6d5cda0af8]) successfully deleted
I20260929 20:57:50.069588 19112 catalog_manager.cc:4841] TS 2027f84aa41a49c38fb6c3a900d1573e (127.0.97.2:45655): tablet 4e790a3057ec49d1af1384cb7822c51a (table test-table [id=a648c184fbae4500a0457c6d5cda0af8]) successfully deleted
I20260929 20:57:50.152709 19615 raft_consensus.cc:2258] T 4e790a3057ec49d1af1384cb7822c51a P 2fc1fbfc91964c7eaa3b3c2c8cf09a5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:57:50.153050 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 19186
I20260929 20:57:50.167513 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 19317
W20260929 20:57:50.167507 19114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39569 (error 108)
I20260929 20:57:50.169354 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:50.168855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39569 (local address 127.0.0.1:45010)
0929 20:57:50.169049 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:50.169171 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260929 20:57:50.169612 19114 catalog_manager.cc:4584] TS 2fc1fbfc91964c7eaa3b3c2c8cf09a5e (127.0.97.1:39569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e790a3057ec49d1af1384cb7822c51a: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39569: connect: Connection refused (error 111)
I20260929 20:57:50.174427 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:50.168367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.0.97.2:45655)
0929 20:57:50.168549 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:50.168555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:50.168674 (+ 119us) server_negotiation.cc:407] Connection header received
0929 20:57:50.168893 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:50.168899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:50.169024 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:50.169134 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:50.169664 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:50.170485 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:50.171752 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:50.172134 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:50.172219 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:50.173143 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:50.173150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:50.173163 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:50.173728 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:50.173771 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:50.173790 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:50.173869 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:50.173950 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 20:57:50.174033 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:57:50.175098 19439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:50.223213 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:50.222670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39569 (local address 127.0.0.1:45022)
0929 20:57:50.222904 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:50.223018 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 20:57:50.224293 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:50.223887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39569 (local address 127.0.0.1:45036)
0929 20:57:50.224054 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:50.224141 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 20:57:50.262648 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 19317
I20260929 20:57:50.276417 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 19447
I20260929 20:57:50.283757 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:50.277260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.0.97.3:33231)
0929 20:57:50.277467 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:50.277472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:50.277492 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:57:50.277588 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:50.277592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:50.277687 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:50.277771 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:50.278113 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:50.279154 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:50.280403 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:50.281017 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:50.281132 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:50.282257 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:50.282271 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:50.282291 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:50.283019 (+ 728us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:50.283079 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:50.283103 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:50.283233 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:50.283347 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 20:57:50.283460 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:57:50.284433 19569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:50.285240 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:50.284722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39569 (local address 127.0.0.1:45038)
0929 20:57:50.285004 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:50.285087 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260929 20:57:50.286386 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:50.285971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39569 (local address 127.0.0.1:45044)
0929 20:57:50.286178 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:50.286276 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 20:57:50.367283 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:50.366721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39569 (local address 127.0.0.1:45054)
0929 20:57:50.366973 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:50.367087 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 20:57:50.368521 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:50.368055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39569 (local address 127.0.0.1:45068)
0929 20:57:50.368269 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:50.368358 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260929 20:57:50.373641 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 19447
I20260929 20:57:50.386881 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 19096
I20260929 20:57:50.388028 19157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:50.496538 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 19096
2026-09-29T20:57:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2690 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:57:50 dist-test-slave-m9g1 krb5kdc[19626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:57:50 dist-test-slave-m9g1 krb5kdc[19626](info): set up 2 sockets
Sep 29 20:57:50 dist-test-slave-m9g1 krb5kdc[19626](info): commencing operation
krb5kdc: starting...
W20260929 20:57:51.518254 388 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:57:51 dist-test-slave-m9g1 krb5kdc[19626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:57:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:57:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:57:51.755318 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43489
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:57:52.075981 19639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:52.076293 19639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:52.076433 19639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:52.083887 19639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:57:52.084024 19639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:52.084103 19639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:52.084180 19639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:57:52.084239 19639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:57:52.094180 19639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:52.095489 19639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:52.096752 19639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:52.107934 19644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:52.107924 19645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:52.107877 19647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:52.108618 19639 server_base.cc:1034] running on GCE node
I20260929 20:57:52.109366 19639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:52.110809 19639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:52.112022 19639 hybrid_clock.cc:648] HybridClock initialized: now 1790715472111969 us; error 44 us; skew 500 ppm
I20260929 20:57:52.112353 19639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:57:52 dist-test-slave-m9g1 krb5kdc[19626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:57:52.120692 19639 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:57:52.122243 19639 webserver.cc:460] Webserver started at http://127.0.97.62:33101/ using document root <none> and password file <none>
I20260929 20:57:52.122951 19639 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:52.123049 19639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:52.123312 19639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:52.126319 19639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "80f7cbee26274ee88066bc492389ef24"
format_stamp: "Formatted at 2026-09-29 20:57:52 on dist-test-slave-m9g1"
server_key: "c19a15ff96dacd794ddd587e9f77f121"
server_key_iv: "be3741aafb2c94dd2187ed55209ca523"
server_key_version: "encryptionkey@0"
I20260929 20:57:52.127074 19639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "80f7cbee26274ee88066bc492389ef24"
format_stamp: "Formatted at 2026-09-29 20:57:52 on dist-test-slave-m9g1"
server_key: "c19a15ff96dacd794ddd587e9f77f121"
server_key_iv: "be3741aafb2c94dd2187ed55209ca523"
server_key_version: "encryptionkey@0"
I20260929 20:57:52.132089 19639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 20:57:52.135385 19654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:52.136746 19639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:57:52.136900 19639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "80f7cbee26274ee88066bc492389ef24"
format_stamp: "Formatted at 2026-09-29 20:57:52 on dist-test-slave-m9g1"
server_key: "c19a15ff96dacd794ddd587e9f77f121"
server_key_iv: "be3741aafb2c94dd2187ed55209ca523"
server_key_version: "encryptionkey@0"
I20260929 20:57:52.137084 19639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:52.151520 19639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:52.155241 19639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:52.155493 19639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:52.172420 19639 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43489
I20260929 20:57:52.172439 19705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43489 every 8 connection(s)
I20260929 20:57:52.173728 19639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:57:52.177356 19706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:52.177744 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 19639
I20260929 20:57:52.177924 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:57:52.178476 388 external_mini_cluster.cc:1428] Setting key ebb03fd5bcf0e75367f77254b55ddb0b
I20260929 20:57:52.191162 19706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24: Bootstrap starting.
Sep 29 20:57:52 dist-test-slave-m9g1 krb5kdc[19626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:57:52.194783 19706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:52.195796 19706 log.cc:824] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:52.198963 19706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24: No bootstrap required, opened a new log
I20260929 20:57:52.202589 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:52.180330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49562 (local address 127.0.97.62:43489)
0929 20:57:52.181122 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:52.181134 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:52.181172 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:57:52.183441 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:52.183468 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:52.183879 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:52.184219 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:52.186038 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:52.187443 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:52.189286 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:52.189936 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:52.190060 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:52.194572 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:52.194581 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:52.194595 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:52.194634 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:52.198040 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:52.198763 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:52.198771 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:52.198779 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:52.199003 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:52.199415 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:52.199423 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:52.199427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:52.200109 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:52.200400 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:52.200781 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 20:57:52.201234 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":350,"threads_started":1}
I20260929 20:57:52.207434 19706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f7cbee26274ee88066bc492389ef24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43489 } }
I20260929 20:57:52.207772 19706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:52.207846 19706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f7cbee26274ee88066bc492389ef24, State: Initialized, Role: FOLLOWER
I20260929 20:57:52.208312 19706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f7cbee26274ee88066bc492389ef24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43489 } }
I20260929 20:57:52.208511 19706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:57:52.208606 19706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:57:52.208717 19706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:52.210826 19706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f7cbee26274ee88066bc492389ef24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43489 } }
I20260929 20:57:52.211292 19706 leader_election.cc:302] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80f7cbee26274ee88066bc492389ef24; no voters:
I20260929 20:57:52.211901 19706 leader_election.cc:288] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:57:52.212028 19711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:57:52.212618 19711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24 [term 1 LEADER]: Becoming Leader. State: Replica: 80f7cbee26274ee88066bc492389ef24, State: Running, Role: LEADER
I20260929 20:57:52.213121 19711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f7cbee26274ee88066bc492389ef24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43489 } }
I20260929 20:57:52.213774 19706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:57:52.215620 19713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80f7cbee26274ee88066bc492389ef24. Latest consensus state: current_term: 1 leader_uuid: "80f7cbee26274ee88066bc492389ef24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f7cbee26274ee88066bc492389ef24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43489 } } }
I20260929 20:57:52.216095 19713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24 [sys.catalog]: This master's current role is: LEADER
I20260929 20:57:52.216917 19712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80f7cbee26274ee88066bc492389ef24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f7cbee26274ee88066bc492389ef24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43489 } } }
I20260929 20:57:52.217278 19712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24 [sys.catalog]: This master's current role is: LEADER
I20260929 20:57:52.219169 19720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:57:52.224720 19720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:57:52.233400 19720 catalog_manager.cc:1347] Generated new cluster ID: e9e2378840334efa8714fab592ad6a21
I20260929 20:57:52.233513 19720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:57:52.249740 19720 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:57:52.251735 19720 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:57:52.262050 19720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24: Generated new TSK 0
I20260929 20:57:52.263208 19720 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:57:52.317734 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43489
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:57:52.646023 19732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:52.646509 19732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:52.646641 19732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:52.654109 19732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:52.654245 19732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:52.654385 19732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:57:52.664266 19732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:52.665541 19732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:52.666850 19732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:52.680157 19738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:52.680136 19737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:52.680102 19740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:52.681010 19732 server_base.cc:1034] running on GCE node
I20260929 20:57:52.681643 19732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:52.682687 19732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:52.683909 19732 hybrid_clock.cc:648] HybridClock initialized: now 1790715472683837 us; error 60 us; skew 500 ppm
I20260929 20:57:52.684247 19732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:57:52 dist-test-slave-m9g1 krb5kdc[19626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:57:52.687546 19732 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:57:52.689102 19732 webserver.cc:460] Webserver started at http://127.0.97.1:42359/ using document root <none> and password file <none>
I20260929 20:57:52.689843 19732 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:52.689981 19732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:52.690339 19732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:52.693437 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "cb7d01309b7a471b9bb33f0fd1b31fe4"
format_stamp: "Formatted at 2026-09-29 20:57:52 on dist-test-slave-m9g1"
server_key: "6fcfb8c1c0a08b2a8cea5428116db526"
server_key_iv: "58a9909e482c72b5fa6c62c61894e703"
server_key_version: "encryptionkey@0"
I20260929 20:57:52.694239 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cb7d01309b7a471b9bb33f0fd1b31fe4"
format_stamp: "Formatted at 2026-09-29 20:57:52 on dist-test-slave-m9g1"
server_key: "6fcfb8c1c0a08b2a8cea5428116db526"
server_key_iv: "58a9909e482c72b5fa6c62c61894e703"
server_key_version: "encryptionkey@0"
I20260929 20:57:52.699397 19732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:57:52.703054 19747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:52.704617 19732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:57:52.704756 19732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "cb7d01309b7a471b9bb33f0fd1b31fe4"
format_stamp: "Formatted at 2026-09-29 20:57:52 on dist-test-slave-m9g1"
server_key: "6fcfb8c1c0a08b2a8cea5428116db526"
server_key_iv: "58a9909e482c72b5fa6c62c61894e703"
server_key_version: "encryptionkey@0"
I20260929 20:57:52.704921 19732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:52.720393 19732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:52.724555 19732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:52.724797 19732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:52.725739 19732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:52.727072 19732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:52.727142 19732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:52.727229 19732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:52.727274 19732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:52.757588 19732 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41893
I20260929 20:57:52.757623 19859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41893 every 8 connection(s)
I20260929 20:57:52.758869 19732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:57:52.764541 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 19732
I20260929 20:57:52.764742 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:57:52.765237 388 external_mini_cluster.cc:1428] Setting key 45e592ebea8aa100a6c07e023b479f0c
Sep 29 20:57:52 dist-test-slave-m9g1 krb5kdc[19626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:57:52.780232 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:52.761397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46355 (local address 127.0.97.62:43489)
0929 20:57:52.761840 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:52.761845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:52.762869 (+ 1024us) server_negotiation.cc:407] Connection header received
0929 20:57:52.764947 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:52.764952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:52.765030 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:52.765117 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:52.766956 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:52.767781 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:52.770006 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:52.770449 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:52.770550 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:52.774478 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:52.774483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:52.774487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:52.774523 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:52.777015 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:52.777708 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:52.777713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:52.777715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:52.777832 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:52.778480 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:52.778484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:52.778486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:52.778775 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:52.778918 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:52.779653 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 20:57:52.779880 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":234,"threads_started":1}
I20260929 20:57:52.781280 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:52.761979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43489 (local address 127.0.97.1:46355)
0929 20:57:52.762719 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:52.762757 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:52.764586 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:52.765408 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:52.765421 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:52.766102 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:57:52.766785 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:52.766804 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:52.767972 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:52.767978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:52.769831 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:52.769844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:52.770658 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:52.770664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:52.770945 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:52.771606 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:57:52.771632 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:57:52.774245 (+ 2613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:57:52.777153 (+ 2908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:57:52.777162 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:57:52.777181 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:57:52.777586 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:57:52.777939 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:57:52.777944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:57:52.777948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:57:52.778353 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:57:52.778951 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:57:52.778960 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:57:52.779286 (+ 326us) client_negotiation.cc:769] Sending connection context
0929 20:57:52.779533 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 20:57:52.780080 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":299,"threads_started":1}
I20260929 20:57:52.783075 19860 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43489
I20260929 20:57:52.783372 19860 heartbeater.cc:464] Registering TS with master...
I20260929 20:57:52.784124 19860 heartbeater.cc:511] Master 127.0.97.62:43489 requested a full tablet report, sending...
I20260929 20:57:52.786543 19671 ts_manager.cc:194] Registered new tserver with Master: cb7d01309b7a471b9bb33f0fd1b31fe4 (127.0.97.1:41893)
I20260929 20:57:52.788630 19671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46355
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:57:52.819344 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43489
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:57:53.142670 19866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:53.142987 19866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:53.143129 19866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:53.150424 19866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:53.150539 19866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:53.150688 19866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:57:53.160379 19866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:53.161701 19866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:53.163023 19866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:53.175567 19874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:53.175604 19871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:53.176312 19872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:53.183898 19866 server_base.cc:1034] running on GCE node
I20260929 20:57:53.184623 19866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:53.185711 19866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:53.186957 19866 hybrid_clock.cc:648] HybridClock initialized: now 1790715473186885 us; error 61 us; skew 500 ppm
I20260929 20:57:53.187256 19866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:57:53 dist-test-slave-m9g1 krb5kdc[19626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:57:53.190301 19866 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:57:53.191779 19866 webserver.cc:460] Webserver started at http://127.0.97.2:43419/ using document root <none> and password file <none>
I20260929 20:57:53.192478 19866 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:53.192576 19866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:53.192878 19866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:53.195803 19866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "c901bb2f51ea447b833d1588f5e672ee"
format_stamp: "Formatted at 2026-09-29 20:57:53 on dist-test-slave-m9g1"
server_key: "4ca99b43971c1fe9753f3de899b09f08"
server_key_iv: "cb1c616ace3374c11c7c39bd59908e13"
server_key_version: "encryptionkey@0"
I20260929 20:57:53.196574 19866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c901bb2f51ea447b833d1588f5e672ee"
format_stamp: "Formatted at 2026-09-29 20:57:53 on dist-test-slave-m9g1"
server_key: "4ca99b43971c1fe9753f3de899b09f08"
server_key_iv: "cb1c616ace3374c11c7c39bd59908e13"
server_key_version: "encryptionkey@0"
I20260929 20:57:53.201665 19866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:57:53.205178 19881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:53.206580 19866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:57:53.206736 19866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "c901bb2f51ea447b833d1588f5e672ee"
format_stamp: "Formatted at 2026-09-29 20:57:53 on dist-test-slave-m9g1"
server_key: "4ca99b43971c1fe9753f3de899b09f08"
server_key_iv: "cb1c616ace3374c11c7c39bd59908e13"
server_key_version: "encryptionkey@0"
I20260929 20:57:53.206941 19866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:53.219501 19866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:53.223841 19866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:53.224123 19866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:53.225116 19866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:53.226538 19866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:53.226616 19866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:53.226735 19866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:53.226786 19866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:53.257561 19866 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45689
I20260929 20:57:53.257597 19993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45689 every 8 connection(s)
I20260929 20:57:53.258821 19866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:57:53.262553 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 19866
I20260929 20:57:53.262727 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:57:53.263252 388 external_mini_cluster.cc:1428] Setting key 6683b169bd3635c35f1517c2b39ab522
Sep 29 20:57:53 dist-test-slave-m9g1 krb5kdc[19626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:57:53.280504 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:53.261268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39047 (local address 127.0.97.62:43489)
0929 20:57:53.261485 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:53.261491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:53.262844 (+ 1353us) server_negotiation.cc:407] Connection header received
0929 20:57:53.265119 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:53.265125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:53.265204 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:53.265309 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:53.267328 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.268029 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.270030 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.270489 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.270584 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:53.274569 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:53.274575 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:53.274580 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:53.274625 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:53.277345 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:53.278163 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:53.278167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:53.278170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:53.278347 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:53.278779 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:53.278785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:53.278788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:53.279064 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:53.279195 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:53.279904 (+ 709us) server_negotiation.cc:299] Negotiation successful
0929 20:57:53.280151 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:57:53.281507 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:53.261775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43489 (local address 127.0.97.2:39047)
0929 20:57:53.262653 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:53.262712 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:53.264799 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:53.265571 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:53.265583 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:53.266362 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:57:53.267160 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:53.267187 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.268178 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.268183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:53.269896 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:53.269919 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.270694 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.270700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:53.270982 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:53.271674 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:57:53.271701 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:57:53.274315 (+ 2614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:57:53.277502 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:57:53.277512 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:57:53.277531 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:57:53.278025 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:57:53.278457 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:57:53.278462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:57:53.278466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:57:53.278661 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:57:53.279188 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:57:53.279197 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:57:53.279546 (+ 349us) client_negotiation.cc:769] Sending connection context
0929 20:57:53.279786 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 20:57:53.280361 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":359,"threads_started":1}
I20260929 20:57:53.282987 19994 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43489
I20260929 20:57:53.283291 19994 heartbeater.cc:464] Registering TS with master...
I20260929 20:57:53.284027 19994 heartbeater.cc:511] Master 127.0.97.62:43489 requested a full tablet report, sending...
I20260929 20:57:53.285612 19671 ts_manager.cc:194] Registered new tserver with Master: c901bb2f51ea447b833d1588f5e672ee (127.0.97.2:45689)
I20260929 20:57:53.286460 19671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39047
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:57:53.316717 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43489
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:57:53.631734 19999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:53.632148 19999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:53.632306 19999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:53.639640 19999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:53.639750 19999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:53.639899 19999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:57:53.649672 19999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:53.650961 19999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:53.652357 19999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:53.665051 20007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:53.665031 20004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:53.665091 20005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:53.665846 19999 server_base.cc:1034] running on GCE node
I20260929 20:57:53.666548 19999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:53.667496 19999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:53.668701 19999 hybrid_clock.cc:648] HybridClock initialized: now 1790715473668647 us; error 44 us; skew 500 ppm
I20260929 20:57:53.669140 19999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:57:53 dist-test-slave-m9g1 krb5kdc[19626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:57:53.672350 19999 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:57:53.673785 19999 webserver.cc:460] Webserver started at http://127.0.97.3:45859/ using document root <none> and password file <none>
I20260929 20:57:53.674535 19999 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:53.674649 19999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:53.674952 19999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:53.677887 19999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "48ed10798609491e9b0d5521c56ec2c6"
format_stamp: "Formatted at 2026-09-29 20:57:53 on dist-test-slave-m9g1"
server_key: "d4a34080b7caa59d539000ce85b813b1"
server_key_iv: "c8aeece93eec378aa991fb9b7a545b13"
server_key_version: "encryptionkey@0"
I20260929 20:57:53.678767 19999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "48ed10798609491e9b0d5521c56ec2c6"
format_stamp: "Formatted at 2026-09-29 20:57:53 on dist-test-slave-m9g1"
server_key: "d4a34080b7caa59d539000ce85b813b1"
server_key_iv: "c8aeece93eec378aa991fb9b7a545b13"
server_key_version: "encryptionkey@0"
I20260929 20:57:53.683758 19999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:57:53.687232 20014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:53.688702 19999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:57:53.688885 19999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "48ed10798609491e9b0d5521c56ec2c6"
format_stamp: "Formatted at 2026-09-29 20:57:53 on dist-test-slave-m9g1"
server_key: "d4a34080b7caa59d539000ce85b813b1"
server_key_iv: "c8aeece93eec378aa991fb9b7a545b13"
server_key_version: "encryptionkey@0"
I20260929 20:57:53.689083 19999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:53.721570 19999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:53.725543 19999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:53.725800 19999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:53.726784 19999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:53.728051 19999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:53.728125 19999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:53.728237 19999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:53.728302 19999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:53.757748 19999 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39667
I20260929 20:57:53.757825 20126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39667 every 8 connection(s)
I20260929 20:57:53.759040 19999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:57:53.759449 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 19999
I20260929 20:57:53.759613 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:57:53.760123 388 external_mini_cluster.cc:1428] Setting key fe896aaa9de08fb779ba2ae4af92399b
Sep 29 20:57:53 dist-test-slave-m9g1 krb5kdc[19626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:57:53.778733 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:53.762547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42319 (local address 127.0.97.62:43489)
0929 20:57:53.763278 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:53.763284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:53.763929 (+ 645us) server_negotiation.cc:407] Connection header received
0929 20:57:53.765629 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:53.765634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:53.765697 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:53.765773 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:53.767197 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.767965 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.769422 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.769801 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.769900 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:53.773158 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:53.773163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:53.773166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:53.773198 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:53.776135 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:53.776766 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:53.776770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:53.776772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:53.776924 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:53.777263 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:53.777266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:53.777268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:53.777533 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:53.777644 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:53.778325 (+ 681us) server_negotiation.cc:299] Negotiation successful
0929 20:57:53.778495 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616}
I20260929 20:57:53.779633 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:53.763070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43489 (local address 127.0.97.3:42319)
0929 20:57:53.763794 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:53.763829 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:53.765395 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:53.765955 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:53.765964 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:53.766516 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:57:53.767054 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:53.767072 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.768084 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.768087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:53.769285 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:53.769296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.769975 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.769979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:53.770172 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:53.770845 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:57:53.770874 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:57:53.772963 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:57:53.776257 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:57:53.776264 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:57:53.776278 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:57:53.776674 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:57:53.777001 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:57:53.777004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:57:53.777007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:57:53.777175 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:57:53.777625 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:57:53.777630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:57:53.777963 (+ 333us) client_negotiation.cc:769] Sending connection context
0929 20:57:53.778217 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 20:57:53.778606 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":320,"threads_started":1}
I20260929 20:57:53.780934 20127 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43489
I20260929 20:57:53.781198 20127 heartbeater.cc:464] Registering TS with master...
I20260929 20:57:53.781867 20127 heartbeater.cc:511] Master 127.0.97.62:43489 requested a full tablet report, sending...
I20260929 20:57:53.783308 19671 ts_manager.cc:194] Registered new tserver with Master: 48ed10798609491e9b0d5521c56ec2c6 (127.0.97.3:39667)
I20260929 20:57:53.784080 19671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42319
I20260929 20:57:53.788059 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:57:53.791918 19860 heartbeater.cc:503] Master 127.0.97.62:43489 was elected leader, sending a full tablet report...
I20260929 20:57:53.803812 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:53.792046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49564 (local address 127.0.97.62:43489)
0929 20:57:53.792236 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:53.792242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:53.792660 (+ 418us) server_negotiation.cc:407] Connection header received
0929 20:57:53.792782 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:53.792787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:53.792877 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:53.792973 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:53.794473 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.795184 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.796475 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.796857 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.796948 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:53.799086 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:53.799091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:53.799094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:53.799125 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:53.801534 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:53.802133 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:53.802136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:53.802138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:53.802291 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:53.802564 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:53.802567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:53.802569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:53.802854 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:53.803053 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:53.803315 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 20:57:53.803538 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:57:53.807713 19671 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:57:53.814570 19671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:57:53.838529 20140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:53.825804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41893 (local address 127.0.0.1:40534)
0929 20:57:53.828405 (+ 2601us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:53.828442 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:53.828631 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:53.829705 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:53.829713 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:53.829746 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:53.830170 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:53.830215 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.832319 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.832326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:53.835006 (+ 2680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:53.835024 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.837410 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.837417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:53.837660 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:53.837715 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 20:57:53.837846 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 20:57:53.838067 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2368,"thread_start_us":2218,"threads_started":1}
I20260929 20:57:53.839352 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:53.825956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.0.97.1:41893)
0929 20:57:53.828006 (+ 2050us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:53.828014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:53.828829 (+ 815us) server_negotiation.cc:407] Connection header received
0929 20:57:53.828938 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:53.828947 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:53.829226 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:53.829401 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:53.830399 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.832138 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.835217 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.836845 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.836975 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:53.837257 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:53.838791 (+ 1534us) server_negotiation.cc:299] Negotiation successful
0929 20:57:53.839005 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1896,"thread_start_us":1728,"threads_started":1}
I20260929 20:57:53.844151 19795 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8b528c9ed64699b94124c41cb6ff2d (DEFAULT_TABLE table=test-table [id=29ce8c720f1741c5b4f4b40af3b5465b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:57:53.845690 19795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8b528c9ed64699b94124c41cb6ff2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:53.849115 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:53.825851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39667 (local address 127.0.0.1:44992)
0929 20:57:53.839882 (+ 14031us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:53.839903 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:53.840038 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:53.841345 (+ 1307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:53.841350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:53.841461 (+ 111us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:53.841854 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:53.841867 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.844077 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.844083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:53.846330 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:53.846348 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.848424 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.848429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:53.848550 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:53.848573 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:57:53.848650 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 20:57:53.848782 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13883,"thread_start_us":14275,"threads_started":1}
I20260929 20:57:53.849212 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:53.826393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44992 (local address 127.0.97.3:39667)
0929 20:57:53.827174 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:53.827184 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:53.840012 (+ 12828us) server_negotiation.cc:407] Connection header received
0929 20:57:53.840488 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:53.840499 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:53.840816 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:53.841030 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:53.842102 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.843871 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.846542 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.847906 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.848005 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:53.848238 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:53.848727 (+ 489us) server_negotiation.cc:299] Negotiation successful
0929 20:57:53.848900 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":413,"threads_started":1}
I20260929 20:57:53.853513 20140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:53.827852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45689 (local address 127.0.0.1:45648)
0929 20:57:53.840116 (+ 12264us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:53.840135 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:53.840238 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:53.847925 (+ 7687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:53.847930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:53.847960 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:53.848200 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:53.848211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.850267 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.850272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:53.851421 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:53.851434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.852804 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.852810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:53.852981 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:53.853015 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:57:53.853096 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 20:57:53.853231 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12152,"thread_start_us":18939,"threads_started":1}
I20260929 20:57:53.853609 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:53.828396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45648 (local address 127.0.97.2:45689)
0929 20:57:53.847134 (+ 18738us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:53.847143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:53.847172 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:57:53.847265 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:53.847272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:53.847527 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:53.848377 (+ 850us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:53.848603 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.850094 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.851563 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:53.852611 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:53.852776 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:53.852971 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:53.853144 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 20:57:53.853307 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18553,"thread_start_us":19266,"threads_started":1}
I20260929 20:57:53.853338 20062 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8b528c9ed64699b94124c41cb6ff2d (DEFAULT_TABLE table=test-table [id=29ce8c720f1741c5b4f4b40af3b5465b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:57:53.854904 20062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8b528c9ed64699b94124c41cb6ff2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:53.857178 19929 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8b528c9ed64699b94124c41cb6ff2d (DEFAULT_TABLE table=test-table [id=29ce8c720f1741c5b4f4b40af3b5465b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:57:53.858301 19929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8b528c9ed64699b94124c41cb6ff2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:53.860589 20145 tablet_bootstrap.cc:492] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4: Bootstrap starting.
I20260929 20:57:53.864168 20145 tablet_bootstrap.cc:654] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:53.865460 20145 log.cc:824] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:53.867801 20147 tablet_bootstrap.cc:492] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6: Bootstrap starting.
I20260929 20:57:53.868546 20145 tablet_bootstrap.cc:492] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4: No bootstrap required, opened a new log
I20260929 20:57:53.868847 20145 ts_tablet_manager.cc:1397] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260929 20:57:53.870685 20147 tablet_bootstrap.cc:654] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:53.871536 20148 tablet_bootstrap.cc:492] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee: Bootstrap starting.
I20260929 20:57:53.871914 20147 log.cc:824] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:53.874833 20148 tablet_bootstrap.cc:654] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:53.875797 20147 tablet_bootstrap.cc:492] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6: No bootstrap required, opened a new log
I20260929 20:57:53.876063 20147 ts_tablet_manager.cc:1397] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260929 20:57:53.876077 20148 log.cc:824] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:53.877593 20145 raft_consensus.cc:348] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7d01309b7a471b9bb33f0fd1b31fe4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41893 } } peers { permanent_uuid: "48ed10798609491e9b0d5521c56ec2c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39667 } } peers { permanent_uuid: "c901bb2f51ea447b833d1588f5e672ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45689 } }
I20260929 20:57:53.877934 20145 raft_consensus.cc:374] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:53.878023 20145 raft_consensus.cc:729] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb7d01309b7a471b9bb33f0fd1b31fe4, State: Initialized, Role: FOLLOWER
I20260929 20:57:53.878609 20145 consensus_queue.cc:260] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7d01309b7a471b9bb33f0fd1b31fe4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41893 } } peers { permanent_uuid: "48ed10798609491e9b0d5521c56ec2c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39667 } } peers { permanent_uuid: "c901bb2f51ea447b833d1588f5e672ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45689 } }
I20260929 20:57:53.878985 20148 tablet_bootstrap.cc:492] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee: No bootstrap required, opened a new log
I20260929 20:57:53.879293 20148 ts_tablet_manager.cc:1397] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260929 20:57:53.880206 20145 ts_tablet_manager.cc:1428] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 20:57:53.886147 20147 raft_consensus.cc:348] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7d01309b7a471b9bb33f0fd1b31fe4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41893 } } peers { permanent_uuid: "48ed10798609491e9b0d5521c56ec2c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39667 } } peers { permanent_uuid: "c901bb2f51ea447b833d1588f5e672ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45689 } }
I20260929 20:57:53.886612 20147 raft_consensus.cc:374] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:53.886710 20147 raft_consensus.cc:729] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ed10798609491e9b0d5521c56ec2c6, State: Initialized, Role: FOLLOWER
I20260929 20:57:53.887315 20147 consensus_queue.cc:260] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7d01309b7a471b9bb33f0fd1b31fe4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41893 } } peers { permanent_uuid: "48ed10798609491e9b0d5521c56ec2c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39667 } } peers { permanent_uuid: "c901bb2f51ea447b833d1588f5e672ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45689 } }
I20260929 20:57:53.888461 20127 heartbeater.cc:503] Master 127.0.97.62:43489 was elected leader, sending a full tablet report...
I20260929 20:57:53.889150 20147 ts_tablet_manager.cc:1428] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260929 20:57:53.889139 20148 raft_consensus.cc:348] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7d01309b7a471b9bb33f0fd1b31fe4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41893 } } peers { permanent_uuid: "48ed10798609491e9b0d5521c56ec2c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39667 } } peers { permanent_uuid: "c901bb2f51ea447b833d1588f5e672ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45689 } }
I20260929 20:57:53.889524 20148 raft_consensus.cc:374] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:53.889613 20148 raft_consensus.cc:729] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c901bb2f51ea447b833d1588f5e672ee, State: Initialized, Role: FOLLOWER
I20260929 20:57:53.890079 20148 consensus_queue.cc:260] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7d01309b7a471b9bb33f0fd1b31fe4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41893 } } peers { permanent_uuid: "48ed10798609491e9b0d5521c56ec2c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39667 } } peers { permanent_uuid: "c901bb2f51ea447b833d1588f5e672ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45689 } }
I20260929 20:57:53.891111 19994 heartbeater.cc:503] Master 127.0.97.62:43489 was elected leader, sending a full tablet report...
I20260929 20:57:53.891513 20148 ts_tablet_manager.cc:1428] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260929 20:57:54.010485 20152 raft_consensus.cc:482] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 20:57:54.010706 19995 tablet.cc:2369] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:57:54.010816 20152 raft_consensus.cc:504] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7d01309b7a471b9bb33f0fd1b31fe4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41893 } } peers { permanent_uuid: "48ed10798609491e9b0d5521c56ec2c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39667 } } peers { permanent_uuid: "c901bb2f51ea447b833d1588f5e672ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45689 } }
W20260929 20:57:54.011058 19861 tablet.cc:2369] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:57:54.011622 20128 tablet.cc:2369] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:57:54.012856 20152 leader_election.cc:288] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb7d01309b7a471b9bb33f0fd1b31fe4 (127.0.97.1:41893), c901bb2f51ea447b833d1588f5e672ee (127.0.97.2:45689)
I20260929 20:57:54.019920 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:54.013286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41613 (local address 127.0.97.2:45689)
0929 20:57:54.013653 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:54.013659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:54.013714 (+ 55us) server_negotiation.cc:407] Connection header received
0929 20:57:54.013818 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:54.013824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:54.013905 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:54.013992 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:54.014667 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:54.015720 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:54.018004 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:54.019149 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:54.019289 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:54.019387 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:54.019581 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 20:57:54.019711 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221}
I20260929 20:57:54.019968 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:54.013209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45689 (local address 127.0.97.3:41613)
0929 20:57:54.013497 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:54.013517 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:54.013637 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:54.014249 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:54.014255 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:54.014285 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:54.014487 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:54.014506 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:54.015824 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:54.015828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:54.017794 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:54.017809 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:54.019293 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:54.019297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:54.019436 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:54.019461 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 20:57:54.019535 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 20:57:54.019671 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260929 20:57:54.020330 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:54.013270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51027 (local address 127.0.97.1:41893)
0929 20:57:54.013474 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:54.013482 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:54.014315 (+ 833us) server_negotiation.cc:407] Connection header received
0929 20:57:54.014485 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:54.014490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:54.014566 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:54.014648 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:54.015148 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:54.016073 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:54.018296 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:54.019467 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:54.019579 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:54.019672 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:54.019973 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 20:57:54.020116 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:57:54.020365 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:54.013259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41893 (local address 127.0.97.3:51027)
0929 20:57:54.014215 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:54.014233 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:54.014361 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:54.014671 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:54.014675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:54.014698 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:54.015010 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:54.015019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:54.016205 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:54.016210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:54.018148 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:54.018158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:54.019651 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:54.019657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:54.019815 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:54.019850 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 20:57:54.019934 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 20:57:54.020055 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":3,"thread_start_us":620,"threads_started":1}
I20260929 20:57:54.020965 19949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8b528c9ed64699b94124c41cb6ff2d" candidate_uuid: "48ed10798609491e9b0d5521c56ec2c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c901bb2f51ea447b833d1588f5e672ee" is_pre_election: true
I20260929 20:57:54.021282 19815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8b528c9ed64699b94124c41cb6ff2d" candidate_uuid: "48ed10798609491e9b0d5521c56ec2c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb7d01309b7a471b9bb33f0fd1b31fe4" is_pre_election: true
I20260929 20:57:54.021457 19949 raft_consensus.cc:2454] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48ed10798609491e9b0d5521c56ec2c6 in term 0.
I20260929 20:57:54.021735 19815 raft_consensus.cc:2454] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48ed10798609491e9b0d5521c56ec2c6 in term 0.
I20260929 20:57:54.022243 20017 leader_election.cc:302] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48ed10798609491e9b0d5521c56ec2c6, c901bb2f51ea447b833d1588f5e672ee; no voters:
I20260929 20:57:54.022631 20152 raft_consensus.cc:2781] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:57:54.022785 20152 raft_consensus.cc:482] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:57:54.022893 20152 raft_consensus.cc:3037] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:54.025012 20152 raft_consensus.cc:504] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7d01309b7a471b9bb33f0fd1b31fe4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41893 } } peers { permanent_uuid: "48ed10798609491e9b0d5521c56ec2c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39667 } } peers { permanent_uuid: "c901bb2f51ea447b833d1588f5e672ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45689 } }
I20260929 20:57:54.025555 20152 leader_election.cc:288] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [CANDIDATE]: Term 1 election: Requested vote from peers cb7d01309b7a471b9bb33f0fd1b31fe4 (127.0.97.1:41893), c901bb2f51ea447b833d1588f5e672ee (127.0.97.2:45689)
I20260929 20:57:54.026000 19815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8b528c9ed64699b94124c41cb6ff2d" candidate_uuid: "48ed10798609491e9b0d5521c56ec2c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb7d01309b7a471b9bb33f0fd1b31fe4"
I20260929 20:57:54.026041 19949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8b528c9ed64699b94124c41cb6ff2d" candidate_uuid: "48ed10798609491e9b0d5521c56ec2c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c901bb2f51ea447b833d1588f5e672ee"
I20260929 20:57:54.026242 19815 raft_consensus.cc:3037] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:54.026253 19949 raft_consensus.cc:3037] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:54.029353 19949 raft_consensus.cc:2454] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48ed10798609491e9b0d5521c56ec2c6 in term 1.
I20260929 20:57:54.029345 19815 raft_consensus.cc:2454] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48ed10798609491e9b0d5521c56ec2c6 in term 1.
I20260929 20:57:54.029769 20017 leader_election.cc:302] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48ed10798609491e9b0d5521c56ec2c6, c901bb2f51ea447b833d1588f5e672ee; no voters:
I20260929 20:57:54.030097 20152 raft_consensus.cc:2781] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:57:54.030553 20152 raft_consensus.cc:686] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [term 1 LEADER]: Becoming Leader. State: Replica: 48ed10798609491e9b0d5521c56ec2c6, State: Running, Role: LEADER
I20260929 20:57:54.031005 20152 consensus_queue.cc:237] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7d01309b7a471b9bb33f0fd1b31fe4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41893 } } peers { permanent_uuid: "48ed10798609491e9b0d5521c56ec2c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39667 } } peers { permanent_uuid: "c901bb2f51ea447b833d1588f5e672ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45689 } }
I20260929 20:57:54.035499 19670 catalog_manager.cc:5495] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48ed10798609491e9b0d5521c56ec2c6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "48ed10798609491e9b0d5521c56ec2c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7d01309b7a471b9bb33f0fd1b31fe4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48ed10798609491e9b0d5521c56ec2c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c901bb2f51ea447b833d1588f5e672ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45689 } health_report { overall_health: UNKNOWN } } }
I20260929 20:57:54.072471 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:54.067358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44994 (local address 127.0.97.3:39667)
0929 20:57:54.067529 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:54.067534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:54.067593 (+ 59us) server_negotiation.cc:407] Connection header received
0929 20:57:54.067767 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:54.067771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:54.067850 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:54.068009 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:57:54.068280 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:54.069009 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:54.070426 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:54.070839 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:54.070962 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:54.071404 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:57:54.071959 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:57:54.072075 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:54.072193 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 20:57:54.072265 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:57:54.080598 19815 raft_consensus.cc:1261] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4 [term 1 FOLLOWER]: Refusing update from remote peer 48ed10798609491e9b0d5521c56ec2c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:57:54.080598 19949 raft_consensus.cc:1261] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee [term 1 FOLLOWER]: Refusing update from remote peer 48ed10798609491e9b0d5521c56ec2c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:57:54.081461 20155 consensus_queue.cc:1035] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb7d01309b7a471b9bb33f0fd1b31fe4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:57:54.081802 20152 consensus_queue.cc:1035] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c901bb2f51ea447b833d1588f5e672ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:57:54.093052 20158 mvcc.cc:205] Tried to move back new op lower bound from 7334770581824708608 to 7334770581634953216. Current Snapshot: MvccSnapshot[applied={T|T < 7334770581824708608}]
I20260929 20:57:54.106805 19670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:57:54.107046 19670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:57:54.110452 19670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80f7cbee26274ee88066bc492389ef24: Sending DeleteTablet for 3 replicas of tablet 8a8b528c9ed64699b94124c41cb6ff2d
I20260929 20:57:54.111565 20062 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8b528c9ed64699b94124c41cb6ff2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:57:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44992
I20260929 20:57:54.111631 19929 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8b528c9ed64699b94124c41cb6ff2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:57:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45648
I20260929 20:57:54.111846 19795 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8b528c9ed64699b94124c41cb6ff2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:57:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40534
I20260929 20:57:54.115818 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 19732
I20260929 20:57:54.140823 20171 tablet_replica.cc:330] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4: stopping tablet replica
I20260929 20:57:54.142709 20171 raft_consensus.cc:2229] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:57:54.145112 20171 raft_consensus.cc:2258] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:57:54.148448 20169 tablet_replica.cc:330] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6: stopping tablet replica
I20260929 20:57:54.148999 20169 raft_consensus.cc:2229] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:57:54.149740 20169 raft_consensus.cc:2258] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:57:54.153242 20171 ts_tablet_manager.cc:1905] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 20:57:54 dist-test-slave-m9g1 krb5kdc[19626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:57:54.158063 20169 ts_tablet_manager.cc:1905] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:57:54.158943 20170 tablet_replica.cc:330] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee: stopping tablet replica
I20260929 20:57:54.159309 20170 raft_consensus.cc:2229] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:57:54.159750 20170 raft_consensus.cc:2258] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:57:54.160360 20171 ts_tablet_manager.cc:1918] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:57:54.160570 20171 log.cc:1196] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-0/wal/wals/8a8b528c9ed64699b94124c41cb6ff2d
I20260929 20:57:54.161155 20171 ts_tablet_manager.cc:1939] T 8a8b528c9ed64699b94124c41cb6ff2d P cb7d01309b7a471b9bb33f0fd1b31fe4: Deleting consensus metadata
I20260929 20:57:54.161931 20170 ts_tablet_manager.cc:1905] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:57:54.163116 19655 catalog_manager.cc:4841] TS cb7d01309b7a471b9bb33f0fd1b31fe4 (127.0.97.1:41893): tablet 8a8b528c9ed64699b94124c41cb6ff2d (table test-table [id=29ce8c720f1741c5b4f4b40af3b5465b]) successfully deleted
I20260929 20:57:54.168324 20170 ts_tablet_manager.cc:1918] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:57:54.168433 20169 ts_tablet_manager.cc:1918] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:57:54.168478 20170 log.cc:1196] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/8a8b528c9ed64699b94124c41cb6ff2d
I20260929 20:57:54.168596 20169 log.cc:1196] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/8a8b528c9ed64699b94124c41cb6ff2d
I20260929 20:57:54.168964 20170 ts_tablet_manager.cc:1939] T 8a8b528c9ed64699b94124c41cb6ff2d P c901bb2f51ea447b833d1588f5e672ee: Deleting consensus metadata
I20260929 20:57:54.169189 20169 ts_tablet_manager.cc:1939] T 8a8b528c9ed64699b94124c41cb6ff2d P 48ed10798609491e9b0d5521c56ec2c6: Deleting consensus metadata
I20260929 20:57:54.170756 19657 catalog_manager.cc:4841] TS c901bb2f51ea447b833d1588f5e672ee (127.0.97.2:45689): tablet 8a8b528c9ed64699b94124c41cb6ff2d (table test-table [id=29ce8c720f1741c5b4f4b40af3b5465b]) successfully deleted
I20260929 20:57:54.171226 19656 catalog_manager.cc:4841] TS 48ed10798609491e9b0d5521c56ec2c6 (127.0.97.3:39667): tablet 8a8b528c9ed64699b94124c41cb6ff2d (table test-table [id=29ce8c720f1741c5b4f4b40af3b5465b]) successfully deleted
I20260929 20:57:54.175606 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:54.117246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40536 (local address 127.0.97.1:41893)
0929 20:57:54.117442 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:54.117449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:54.138365 (+ 20916us) server_negotiation.cc:407] Connection header received
0929 20:57:54.138919 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:54.138927 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:54.139051 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:54.139131 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:54.143104 (+ 3973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:54.144401 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:54.151351 (+ 6950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:54.152158 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:54.152272 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:54.168739 (+ 16467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:54.168754 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:54.168772 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:54.168818 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:54.172451 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:54.173349 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:54.173364 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:54.173367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:54.173539 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:54.173909 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:54.173922 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:54.173925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:54.174403 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:54.174711 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:54.175024 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 20:57:54.175255 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:57:54.176288 19855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:54.296447 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 19732
I20260929 20:57:54.310657 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 19866
Sep 29 20:57:54 dist-test-slave-m9g1 krb5kdc[19626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:57:54.327050 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:54.311464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.0.97.2:45689)
0929 20:57:54.311662 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:54.311667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:54.311688 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:57:54.311855 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:54.311861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:54.311963 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:54.312070 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:54.313253 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:54.314528 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:54.315861 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:54.316452 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:54.316593 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:54.321165 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:54.321181 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:54.321196 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:54.321237 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:54.324273 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:54.325126 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:54.325132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:54.325135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:54.325252 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:54.325618 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:54.325621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:54.325623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:54.326014 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:54.326263 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:54.326586 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 20:57:54.326756 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 20:57:54.327842 19989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:54.416991 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 19866
I20260929 20:57:54.430948 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 19999
Sep 29 20:57:54 dist-test-slave-m9g1 krb5kdc[19626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:57:54.448870 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:54.431801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45006 (local address 127.0.97.3:39667)
0929 20:57:54.431986 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:54.431990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:54.432007 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:57:54.432213 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:54.432217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:54.432284 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:54.432369 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:54.434065 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:54.434907 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:54.436722 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:54.437178 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:54.437267 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:54.442865 (+ 5598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:54.442880 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:54.442906 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:54.442968 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:54.445880 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:54.446815 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:54.446823 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:54.446825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:54.447003 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:54.447455 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:54.447459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:54.447461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:54.447823 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:54.448049 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:54.448382 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 20:57:54.448559 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 20:57:54.449527 20122 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:54.571561 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 19999
I20260929 20:57:54.589866 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 19639
I20260929 20:57:54.590962 19701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:54.699486 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 19639
2026-09-29T20:57:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4203 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T20:57:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:57:54Z Disabled control of system clock
I20260929 20:57:54.768221 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43595
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:57:55.089823 20180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:55.090117 20180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:55.090250 20180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:55.097415 20180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:57:55.097520 20180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:55.097577 20180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:55.097652 20180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:57:55.097703 20180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:57:55.106693 20180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43595
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:55.107950 20180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:55.109726 20180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:55.120810 20188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:55.120915 20185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:55.120911 20186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:55.121717 20180 server_base.cc:1034] running on GCE node
I20260929 20:57:55.122560 20180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:55.123981 20180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:55.125190 20180 hybrid_clock.cc:648] HybridClock initialized: now 1790715475125139 us; error 43 us; skew 500 ppm
I20260929 20:57:55.125542 20180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:57:55.127769 20180 webserver.cc:460] Webserver started at http://127.0.97.62:34505/ using document root <none> and password file <none>
I20260929 20:57:55.128429 20180 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:55.128518 20180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:55.128814 20180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:55.131714 20180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "d208fe7604704669b695a654c45fca42"
format_stamp: "Formatted at 2026-09-29 20:57:55 on dist-test-slave-m9g1"
server_key: "ba97d2425ba4dfcfdc3d18d199282dd3"
server_key_iv: "e897fdaf4dbee6b72290f7dd931b2ed9"
server_key_version: "encryptionkey@0"
I20260929 20:57:55.132498 20180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "d208fe7604704669b695a654c45fca42"
format_stamp: "Formatted at 2026-09-29 20:57:55 on dist-test-slave-m9g1"
server_key: "ba97d2425ba4dfcfdc3d18d199282dd3"
server_key_iv: "e897fdaf4dbee6b72290f7dd931b2ed9"
server_key_version: "encryptionkey@0"
I20260929 20:57:55.137432 20180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 20:57:55.140654 20194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:55.141929 20180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:57:55.142084 20180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "d208fe7604704669b695a654c45fca42"
format_stamp: "Formatted at 2026-09-29 20:57:55 on dist-test-slave-m9g1"
server_key: "ba97d2425ba4dfcfdc3d18d199282dd3"
server_key_iv: "e897fdaf4dbee6b72290f7dd931b2ed9"
server_key_version: "encryptionkey@0"
I20260929 20:57:55.142346 20180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:55.153940 20180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:55.154747 20180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:55.155025 20180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:55.171535 20180 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43595
I20260929 20:57:55.171551 20245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43595 every 8 connection(s)
I20260929 20:57:55.172837 20180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:57:55.176427 20246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:55.180435 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 20180
I20260929 20:57:55.180599 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:57:55.181097 388 external_mini_cluster.cc:1428] Setting key 90bdf868718ef5e5f61732fbb30207f9
I20260929 20:57:55.187462 20246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42: Bootstrap starting.
I20260929 20:57:55.190948 20246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:55.192179 20246 log.cc:824] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:55.195202 20249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:55.182955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.0.97.62:43595)
0929 20:57:55.183853 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:55.183862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:55.183907 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:57:55.186031 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:55.186052 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:55.186532 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:55.186854 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:55.187419 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:55.188735 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:55.190599 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:55.191306 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:55.191449 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:55.192671 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:55.192679 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:55.192694 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:55.193454 (+ 760us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:55.193568 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:55.193596 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:55.193693 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:55.193945 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 20:57:55.194286 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":353,"threads_started":1}
I20260929 20:57:55.195331 20246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42: No bootstrap required, opened a new log
I20260929 20:57:55.202976 20246 raft_consensus.cc:348] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d208fe7604704669b695a654c45fca42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43595 } }
I20260929 20:57:55.203351 20246 raft_consensus.cc:374] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:55.203445 20246 raft_consensus.cc:729] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d208fe7604704669b695a654c45fca42, State: Initialized, Role: FOLLOWER
I20260929 20:57:55.203985 20246 consensus_queue.cc:260] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d208fe7604704669b695a654c45fca42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43595 } }
I20260929 20:57:55.204209 20246 raft_consensus.cc:388] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:57:55.204329 20246 raft_consensus.cc:482] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:57:55.204476 20246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:55.206678 20246 raft_consensus.cc:504] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d208fe7604704669b695a654c45fca42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43595 } }
I20260929 20:57:55.207165 20246 leader_election.cc:302] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d208fe7604704669b695a654c45fca42; no voters:
I20260929 20:57:55.207715 20246 leader_election.cc:288] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:57:55.207863 20251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:57:55.208529 20251 raft_consensus.cc:686] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42 [term 1 LEADER]: Becoming Leader. State: Replica: d208fe7604704669b695a654c45fca42, State: Running, Role: LEADER
I20260929 20:57:55.209060 20251 consensus_queue.cc:237] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d208fe7604704669b695a654c45fca42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43595 } }
I20260929 20:57:55.209856 20246 sys_catalog.cc:564] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:57:55.211884 20253 sys_catalog.cc:455] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d208fe7604704669b695a654c45fca42. Latest consensus state: current_term: 1 leader_uuid: "d208fe7604704669b695a654c45fca42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d208fe7604704669b695a654c45fca42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43595 } } }
I20260929 20:57:55.212028 20252 sys_catalog.cc:455] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d208fe7604704669b695a654c45fca42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d208fe7604704669b695a654c45fca42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43595 } } }
I20260929 20:57:55.212445 20252 sys_catalog.cc:458] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42 [sys.catalog]: This master's current role is: LEADER
I20260929 20:57:55.213591 20253 sys_catalog.cc:458] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42 [sys.catalog]: This master's current role is: LEADER
I20260929 20:57:55.213764 20260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:57:55.218220 20260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:57:55.226591 20260 catalog_manager.cc:1347] Generated new cluster ID: bd31e2059a5d439c82047e1e90ef48a6
I20260929 20:57:55.226694 20260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:57:55.233269 20260 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:57:55.235019 20260 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:57:55.248813 20260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42: Generated new TSK 0
I20260929 20:57:55.249892 20260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:57:55.254050 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43595
--builtin_ntp_servers=127.0.97.20:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:57:55.573971 20270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:55.574280 20270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:55.574411 20270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:55.581607 20270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:55.581710 20270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:55.581858 20270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:57:55.591064 20270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:55.592406 20270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:55.594240 20270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:55.609764 20276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:55.609819 20278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:55.609795 20275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:55.610764 20270 server_base.cc:1034] running on GCE node
I20260929 20:57:55.611403 20270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:55.612336 20270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:55.613523 20270 hybrid_clock.cc:648] HybridClock initialized: now 1790715475613477 us; error 38 us; skew 500 ppm
I20260929 20:57:55.613811 20270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:57:55.616037 20270 webserver.cc:460] Webserver started at http://127.0.97.1:45393/ using document root <none> and password file <none>
I20260929 20:57:55.616700 20270 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:55.616784 20270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:55.617050 20270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:55.619850 20270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-0/data/instance:
uuid: "cebe4b6ba88846e58f074f058bcf768f"
format_stamp: "Formatted at 2026-09-29 20:57:55 on dist-test-slave-m9g1"
server_key: "734652858804e1401d314fe372f00be7"
server_key_iv: "25e99ebef6a54f7630b49948b5ccd014"
server_key_version: "encryptionkey@0"
I20260929 20:57:55.620597 20270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cebe4b6ba88846e58f074f058bcf768f"
format_stamp: "Formatted at 2026-09-29 20:57:55 on dist-test-slave-m9g1"
server_key: "734652858804e1401d314fe372f00be7"
server_key_iv: "25e99ebef6a54f7630b49948b5ccd014"
server_key_version: "encryptionkey@0"
I20260929 20:57:55.625574 20270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:57:55.629000 20284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:55.630412 20270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:57:55.630575 20270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-0/wal
uuid: "cebe4b6ba88846e58f074f058bcf768f"
format_stamp: "Formatted at 2026-09-29 20:57:55 on dist-test-slave-m9g1"
server_key: "734652858804e1401d314fe372f00be7"
server_key_iv: "25e99ebef6a54f7630b49948b5ccd014"
server_key_version: "encryptionkey@0"
I20260929 20:57:55.630780 20270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:55.644341 20270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:55.645200 20270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:55.645529 20270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:55.646574 20270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:55.648080 20270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:55.648165 20270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:55.648247 20270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:55.648299 20270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:55.677946 20270 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37557
I20260929 20:57:55.677991 20396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37557 every 8 connection(s)
I20260929 20:57:55.679272 20270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:57:55.679971 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 20270
I20260929 20:57:55.680114 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:57:55.680595 388 external_mini_cluster.cc:1428] Setting key 596c78afa22ecb6a371b65c958da21cd
I20260929 20:57:55.683696 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43595
--builtin_ntp_servers=127.0.97.20:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:57:55.692118 20249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:55.682303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48709 (local address 127.0.97.62:43595)
0929 20:57:55.682482 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:55.682488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:55.684080 (+ 1592us) server_negotiation.cc:407] Connection header received
0929 20:57:55.685835 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:55.685841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:55.685927 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:55.686012 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:55.687129 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:55.687828 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:55.689407 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:55.689846 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:55.689983 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:55.691346 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:55.691353 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:55.691358 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:55.691380 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:55.691418 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:55.691433 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:55.691514 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:55.691706 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 20:57:55.691816 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:57:55.693048 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:55.682929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43595 (local address 127.0.97.1:48709)
0929 20:57:55.683940 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:55.683985 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:55.685574 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:55.686298 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:55.686307 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:55.686407 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:57:55.686974 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:55.686991 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:55.687979 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:55.687982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:55.689260 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:55.689273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:55.690089 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:55.690094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:55.690365 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:55.691039 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:57:55.691042 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:57:55.691082 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:57:55.691162 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:57:55.691521 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:57:55.691528 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:57:55.691533 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:57:55.691660 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 20:57:55.691983 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":311,"threads_started":1}
I20260929 20:57:55.694581 20397 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43595
I20260929 20:57:55.694856 20397 heartbeater.cc:464] Registering TS with master...
I20260929 20:57:55.695580 20397 heartbeater.cc:511] Master 127.0.97.62:43595 requested a full tablet report, sending...
I20260929 20:57:55.697710 20211 ts_manager.cc:194] Registered new tserver with Master: cebe4b6ba88846e58f074f058bcf768f (127.0.97.1:37557)
I20260929 20:57:55.699643 20211 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:48709
W20260929 20:57:56.002223 20400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:56.002498 20400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:56.002590 20400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:56.009624 20400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:56.009719 20400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:56.009855 20400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:57:56.019198 20400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:56.020457 20400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:56.022132 20400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:56.035077 20406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:56.035128 20408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:56.035291 20405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:56.035698 20400 server_base.cc:1034] running on GCE node
I20260929 20:57:56.036599 20400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:56.037614 20400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:56.038820 20400 hybrid_clock.cc:648] HybridClock initialized: now 1790715476038760 us; error 50 us; skew 500 ppm
I20260929 20:57:56.039152 20400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:57:56.041581 20400 webserver.cc:460] Webserver started at http://127.0.97.2:36125/ using document root <none> and password file <none>
I20260929 20:57:56.042361 20400 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:56.042465 20400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:56.042783 20400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:56.045804 20400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-1/data/instance:
uuid: "c734dce218aa4f47b72ec4ca76bb4894"
format_stamp: "Formatted at 2026-09-29 20:57:56 on dist-test-slave-m9g1"
server_key: "33b2a8a47337fff0784f148fb3f282bd"
server_key_iv: "c0fa0b2919dcfb7a2cf3db7c3ecfd4fc"
server_key_version: "encryptionkey@0"
I20260929 20:57:56.046725 20400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c734dce218aa4f47b72ec4ca76bb4894"
format_stamp: "Formatted at 2026-09-29 20:57:56 on dist-test-slave-m9g1"
server_key: "33b2a8a47337fff0784f148fb3f282bd"
server_key_iv: "c0fa0b2919dcfb7a2cf3db7c3ecfd4fc"
server_key_version: "encryptionkey@0"
I20260929 20:57:56.052254 20400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:57:56.056104 20414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:56.057652 20400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:57:56.057838 20400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-1/wal
uuid: "c734dce218aa4f47b72ec4ca76bb4894"
format_stamp: "Formatted at 2026-09-29 20:57:56 on dist-test-slave-m9g1"
server_key: "33b2a8a47337fff0784f148fb3f282bd"
server_key_iv: "c0fa0b2919dcfb7a2cf3db7c3ecfd4fc"
server_key_version: "encryptionkey@0"
I20260929 20:57:56.058073 20400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:56.092238 20400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:56.093099 20400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:56.093345 20400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:56.094393 20400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:56.095873 20400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:56.095963 20400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:56.096074 20400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:56.096143 20400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:56.126727 20400 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41891
I20260929 20:57:56.126797 20526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41891 every 8 connection(s)
I20260929 20:57:56.127967 20400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:57:56.129364 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 20400
I20260929 20:57:56.129506 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:57:56.129920 388 external_mini_cluster.cc:1428] Setting key 1998828e591dd5da52653ea599d8a897
I20260929 20:57:56.132959 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43595
--builtin_ntp_servers=127.0.97.20:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:57:56.141719 20249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:56.130716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54241 (local address 127.0.97.62:43595)
0929 20:57:56.130903 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:56.130940 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:56.132714 (+ 1774us) server_negotiation.cc:407] Connection header received
0929 20:57:56.134706 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:56.134711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:56.134791 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:56.134876 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:56.135967 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.136858 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.138984 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.139492 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.139596 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:56.141032 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:56.141038 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:56.141041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:56.141062 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:56.141095 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:56.141108 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:56.141186 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:56.141396 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 20:57:56.141477 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:57:56.143123 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:56.131376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43595 (local address 127.0.97.2:54241)
0929 20:57:56.132568 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:56.132611 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:56.134394 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:56.135150 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:56.135158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:56.135252 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:57:56.135800 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:56.135824 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.137031 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.137037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:56.138851 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:56.138870 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.139707 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.139714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:56.139995 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:56.140709 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:57:56.140711 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:57:56.140753 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:57:56.140836 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:57:56.141201 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:57:56.141210 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:57:56.141218 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:57:56.141389 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 20:57:56.141798 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":554,"threads_started":1}
I20260929 20:57:56.144474 20527 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43595
I20260929 20:57:56.144775 20527 heartbeater.cc:464] Registering TS with master...
I20260929 20:57:56.145637 20527 heartbeater.cc:511] Master 127.0.97.62:43595 requested a full tablet report, sending...
I20260929 20:57:56.147995 20211 ts_manager.cc:194] Registered new tserver with Master: c734dce218aa4f47b72ec4ca76bb4894 (127.0.97.2:41891)
I20260929 20:57:56.148818 20211 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:54241
W20260929 20:57:56.451939 20530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:56.452256 20530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:56.452394 20530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:56.459641 20530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:56.459748 20530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:56.459890 20530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:57:56.469084 20530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:56.470396 20530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:56.472124 20530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:56.484520 20535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:56.484575 20536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:56.485033 20538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:56.485157 20530 server_base.cc:1034] running on GCE node
I20260929 20:57:56.485745 20530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:56.486660 20530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:56.487851 20530 hybrid_clock.cc:648] HybridClock initialized: now 1790715476487799 us; error 38 us; skew 500 ppm
I20260929 20:57:56.488142 20530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:57:56.490305 20530 webserver.cc:460] Webserver started at http://127.0.97.3:44373/ using document root <none> and password file <none>
I20260929 20:57:56.491061 20530 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:56.491151 20530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:56.491451 20530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:56.494249 20530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-2/data/instance:
uuid: "1980636c3ba841749b26ab959fe40d58"
format_stamp: "Formatted at 2026-09-29 20:57:56 on dist-test-slave-m9g1"
server_key: "6de762a336a0ea95705d4c8db0cd62c8"
server_key_iv: "76a01f9fc27bb5d10b3a52cbf899292c"
server_key_version: "encryptionkey@0"
I20260929 20:57:56.495003 20530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1980636c3ba841749b26ab959fe40d58"
format_stamp: "Formatted at 2026-09-29 20:57:56 on dist-test-slave-m9g1"
server_key: "6de762a336a0ea95705d4c8db0cd62c8"
server_key_iv: "76a01f9fc27bb5d10b3a52cbf899292c"
server_key_version: "encryptionkey@0"
I20260929 20:57:56.499835 20530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:57:56.503159 20544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:56.504391 20530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:57:56.504539 20530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-2/wal
uuid: "1980636c3ba841749b26ab959fe40d58"
format_stamp: "Formatted at 2026-09-29 20:57:56 on dist-test-slave-m9g1"
server_key: "6de762a336a0ea95705d4c8db0cd62c8"
server_key_iv: "76a01f9fc27bb5d10b3a52cbf899292c"
server_key_version: "encryptionkey@0"
I20260929 20:57:56.504727 20530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:56.518611 20530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:56.519400 20530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:56.519652 20530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:56.520601 20530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:57:56.522046 20530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:57:56.522137 20530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:56.522265 20530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:57:56.522337 20530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:56.551946 20530 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39255
I20260929 20:57:56.551990 20656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39255 every 8 connection(s)
I20260929 20:57:56.553197 20530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:57:56.558943 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 20530
I20260929 20:57:56.559116 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:57:56.559641 388 external_mini_cluster.cc:1428] Setting key 47cd48891c8ac0bf5a7766a79ae748e2
I20260929 20:57:56.565400 20249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:56.555594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47219 (local address 127.0.97.62:43595)
0929 20:57:56.555746 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:56.555751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:56.557083 (+ 1332us) server_negotiation.cc:407] Connection header received
0929 20:57:56.558902 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:56.558905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:56.558980 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:56.559078 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:56.560378 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.561298 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.563138 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.563536 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.563630 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:56.564801 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:56.564808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:56.564812 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:56.564830 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:56.564861 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:56.564873 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:56.564941 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:56.565109 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 20:57:56.565184 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 20:57:56.566347 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:56.556068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43595 (local address 127.0.97.3:47219)
0929 20:57:56.556928 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:56.556962 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:56.558612 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:56.559332 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:56.559343 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:56.559458 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:57:56.560192 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:56.560212 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.561467 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.561473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:56.563016 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:56.563028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.563712 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.563717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:56.563913 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:56.564530 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:57:56.564533 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:57:56.564572 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:57:56.564643 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:57:56.564945 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:57:56.564951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:57:56.564957 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 20:57:56.565093 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 20:57:56.565358 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":266,"threads_started":1}
I20260929 20:57:56.567525 20657 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43595
I20260929 20:57:56.567752 20657 heartbeater.cc:464] Registering TS with master...
I20260929 20:57:56.568485 20657 heartbeater.cc:511] Master 127.0.97.62:43595 requested a full tablet report, sending...
I20260929 20:57:56.569898 20211 ts_manager.cc:194] Registered new tserver with Master: 1980636c3ba841749b26ab959fe40d58 (127.0.97.3:39255)
I20260929 20:57:56.570717 20211 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:47219
I20260929 20:57:56.574514 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:57:56.584786 20249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:56.579446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.0.97.62:43595)
0929 20:57:56.579746 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:56.579753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:56.579978 (+ 225us) server_negotiation.cc:407] Connection header received
0929 20:57:56.580091 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:56.580094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:56.580163 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:56.580239 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:56.580637 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.581351 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.582749 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.583159 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.583257 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:56.584229 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:56.584234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:56.584237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:56.584254 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:56.584286 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:56.584298 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:56.584363 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:56.584468 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 20:57:56.584539 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260929 20:57:56.588219 20211 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:57:56.594727 20211 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:57:56.616693 20671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:56.606286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54144 (local address 127.0.97.3:39255)
0929 20:57:56.607124 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:56.607132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:56.608473 (+ 1341us) server_negotiation.cc:407] Connection header received
0929 20:57:56.608614 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:56.608622 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:56.608909 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:56.609092 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:56.610087 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.611402 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.614037 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.615285 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.615424 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:56.615712 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:56.616181 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 20:57:56.616348 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":1075,"threads_started":1}
I20260929 20:57:56.616701 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:56.605993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39255 (local address 127.0.0.1:54144)
0929 20:57:56.608336 (+ 2343us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:56.608357 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:56.608483 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:56.609321 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:56.609324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:56.609372 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:56.609859 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:56.609874 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.611539 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.611544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:56.613871 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:56.613883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.615816 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.615821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:56.615966 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:56.615999 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:57:56.616111 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 20:57:56.616307 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2244,"mutex_wait_us":93,"spinlock_wait_cycles":22656,"thread_start_us":492,"threads_started":1}
I20260929 20:57:56.618153 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:56.605671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37557 (local address 127.0.0.1:39026)
0929 20:57:56.607728 (+ 2057us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:56.607822 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:56.608157 (+ 335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:56.609306 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:56.609313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:56.609370 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:56.610065 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:56.610078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.612078 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.612082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:56.614760 (+ 2678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:56.614769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.617370 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.617377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:56.617564 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:56.617594 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:57:56.617676 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 20:57:56.617803 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":181,"spinlock_wait_cycles":8832,"thread_start_us":2444,"threads_started":1}
I20260929 20:57:56.618323 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:56.605865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.0.97.1:37557)
0929 20:57:56.606453 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:56.606463 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:56.607982 (+ 1519us) server_negotiation.cc:407] Connection header received
0929 20:57:56.608416 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:56.608424 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:56.608851 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:56.609045 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:56.610281 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.611908 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.614992 (+ 3084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.616383 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.616528 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:56.616753 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:56.617777 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0929 20:57:56.617955 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":265,"threads_started":1}
I20260929 20:57:56.619648 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:56.607247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41891 (local address 127.0.0.1:39384)
0929 20:57:56.608937 (+ 1690us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:56.608958 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:56.609052 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:56.609673 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:56.609681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:56.609704 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:56.610204 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:56.610216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.612094 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.612098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:56.614752 (+ 2654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:56.614767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.618864 (+ 4097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.618870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:56.619034 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:56.619062 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 20:57:56.619135 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 20:57:56.619260 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1572,"mutex_wait_us":198,"thread_start_us":1414,"threads_started":1}
I20260929 20:57:56.619715 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:56.607576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.0.97.2:41891)
0929 20:57:56.608079 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:56.608086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:56.609169 (+ 1083us) server_negotiation.cc:407] Connection header received
0929 20:57:56.609270 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:56.609276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:56.609483 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:56.609663 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:56.610383 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.611955 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.614992 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.618706 (+ 3714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.618829 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:56.619110 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:56.619191 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 20:57:56.619343 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"spinlock_wait_cycles":20736,"thread_start_us":284,"threads_started":1}
I20260929 20:57:56.620752 20592 tablet_service.cc:1467] Processing CreateTablet for tablet e0352574ec1047879fbc71812822ee17 (DEFAULT_TABLE table=test-table [id=b246b18217dd4fcc91da2d0ba4e7271c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:57:56.622377 20592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0352574ec1047879fbc71812822ee17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:56.623550 20462 tablet_service.cc:1467] Processing CreateTablet for tablet e0352574ec1047879fbc71812822ee17 (DEFAULT_TABLE table=test-table [id=b246b18217dd4fcc91da2d0ba4e7271c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:57:56.625042 20462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0352574ec1047879fbc71812822ee17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:56.625263 20332 tablet_service.cc:1467] Processing CreateTablet for tablet e0352574ec1047879fbc71812822ee17 (DEFAULT_TABLE table=test-table [id=b246b18217dd4fcc91da2d0ba4e7271c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:57:56.626361 20332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0352574ec1047879fbc71812822ee17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:56.636937 20675 tablet_bootstrap.cc:492] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58: Bootstrap starting.
I20260929 20:57:56.639015 20676 tablet_bootstrap.cc:492] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894: Bootstrap starting.
I20260929 20:57:56.641673 20675 tablet_bootstrap.cc:654] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:56.641887 20676 tablet_bootstrap.cc:654] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:56.642889 20676 log.cc:824] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:56.643006 20675 log.cc:824] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:56.645391 20676 tablet_bootstrap.cc:492] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894: No bootstrap required, opened a new log
I20260929 20:57:56.645484 20677 tablet_bootstrap.cc:492] T e0352574ec1047879fbc71812822ee17 P cebe4b6ba88846e58f074f058bcf768f: Bootstrap starting.
I20260929 20:57:56.645644 20676 ts_tablet_manager.cc:1397] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260929 20:57:56.648803 20677 tablet_bootstrap.cc:654] T e0352574ec1047879fbc71812822ee17 P cebe4b6ba88846e58f074f058bcf768f: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:56.650206 20677 log.cc:824] T e0352574ec1047879fbc71812822ee17 P cebe4b6ba88846e58f074f058bcf768f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:56.650913 20675 tablet_bootstrap.cc:492] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58: No bootstrap required, opened a new log
I20260929 20:57:56.651181 20675 ts_tablet_manager.cc:1397] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260929 20:57:56.653326 20677 tablet_bootstrap.cc:492] T e0352574ec1047879fbc71812822ee17 P cebe4b6ba88846e58f074f058bcf768f: No bootstrap required, opened a new log
I20260929 20:57:56.653663 20677 ts_tablet_manager.cc:1397] T e0352574ec1047879fbc71812822ee17 P cebe4b6ba88846e58f074f058bcf768f: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260929 20:57:56.653463 20676 raft_consensus.cc:348] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cebe4b6ba88846e58f074f058bcf768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37557 } } peers { permanent_uuid: "1980636c3ba841749b26ab959fe40d58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39255 } } peers { permanent_uuid: "c734dce218aa4f47b72ec4ca76bb4894" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41891 } }
I20260929 20:57:56.653836 20676 raft_consensus.cc:374] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:56.653944 20676 raft_consensus.cc:729] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c734dce218aa4f47b72ec4ca76bb4894, State: Initialized, Role: FOLLOWER
I20260929 20:57:56.654532 20676 consensus_queue.cc:260] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cebe4b6ba88846e58f074f058bcf768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37557 } } peers { permanent_uuid: "1980636c3ba841749b26ab959fe40d58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39255 } } peers { permanent_uuid: "c734dce218aa4f47b72ec4ca76bb4894" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41891 } }
I20260929 20:57:56.655632 20527 heartbeater.cc:503] Master 127.0.97.62:43595 was elected leader, sending a full tablet report...
I20260929 20:57:56.656348 20676 ts_tablet_manager.cc:1428] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260929 20:57:56.661423 20675 raft_consensus.cc:348] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cebe4b6ba88846e58f074f058bcf768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37557 } } peers { permanent_uuid: "1980636c3ba841749b26ab959fe40d58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39255 } } peers { permanent_uuid: "c734dce218aa4f47b72ec4ca76bb4894" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41891 } }
I20260929 20:57:56.661895 20675 raft_consensus.cc:374] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:56.662002 20675 raft_consensus.cc:729] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1980636c3ba841749b26ab959fe40d58, State: Initialized, Role: FOLLOWER
I20260929 20:57:56.662694 20675 consensus_queue.cc:260] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cebe4b6ba88846e58f074f058bcf768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37557 } } peers { permanent_uuid: "1980636c3ba841749b26ab959fe40d58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39255 } } peers { permanent_uuid: "c734dce218aa4f47b72ec4ca76bb4894" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41891 } }
I20260929 20:57:56.663721 20657 heartbeater.cc:503] Master 127.0.97.62:43595 was elected leader, sending a full tablet report...
I20260929 20:57:56.663849 20677 raft_consensus.cc:348] T e0352574ec1047879fbc71812822ee17 P cebe4b6ba88846e58f074f058bcf768f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cebe4b6ba88846e58f074f058bcf768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37557 } } peers { permanent_uuid: "1980636c3ba841749b26ab959fe40d58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39255 } } peers { permanent_uuid: "c734dce218aa4f47b72ec4ca76bb4894" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41891 } }
I20260929 20:57:56.664314 20677 raft_consensus.cc:374] T e0352574ec1047879fbc71812822ee17 P cebe4b6ba88846e58f074f058bcf768f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:56.664424 20677 raft_consensus.cc:729] T e0352574ec1047879fbc71812822ee17 P cebe4b6ba88846e58f074f058bcf768f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cebe4b6ba88846e58f074f058bcf768f, State: Initialized, Role: FOLLOWER
I20260929 20:57:56.664502 20675 ts_tablet_manager.cc:1428] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 20:57:56.665031 20677 consensus_queue.cc:260] T e0352574ec1047879fbc71812822ee17 P cebe4b6ba88846e58f074f058bcf768f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cebe4b6ba88846e58f074f058bcf768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37557 } } peers { permanent_uuid: "1980636c3ba841749b26ab959fe40d58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39255 } } peers { permanent_uuid: "c734dce218aa4f47b72ec4ca76bb4894" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41891 } }
I20260929 20:57:56.666035 20397 heartbeater.cc:503] Master 127.0.97.62:43595 was elected leader, sending a full tablet report...
I20260929 20:57:56.666658 20677 ts_tablet_manager.cc:1428] T e0352574ec1047879fbc71812822ee17 P cebe4b6ba88846e58f074f058bcf768f: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260929 20:57:56.681998 20398 tablet.cc:2369] T e0352574ec1047879fbc71812822ee17 P cebe4b6ba88846e58f074f058bcf768f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:57:56.714291 20681 raft_consensus.cc:482] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:57:56.714644 20681 raft_consensus.cc:504] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cebe4b6ba88846e58f074f058bcf768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37557 } } peers { permanent_uuid: "1980636c3ba841749b26ab959fe40d58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39255 } } peers { permanent_uuid: "c734dce218aa4f47b72ec4ca76bb4894" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41891 } }
I20260929 20:57:56.716689 20681 leader_election.cc:288] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cebe4b6ba88846e58f074f058bcf768f (127.0.97.1:37557), 1980636c3ba841749b26ab959fe40d58 (127.0.97.3:39255)
I20260929 20:57:56.724234 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:56.717450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40021 (local address 127.0.97.1:37557)
0929 20:57:56.717652 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:56.717659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:56.717951 (+ 292us) server_negotiation.cc:407] Connection header received
0929 20:57:56.718045 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:56.718051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:56.718146 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:56.718269 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:56.719186 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.719868 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.722374 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.723421 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.723530 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:56.723620 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:56.723883 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 20:57:56.724001 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:57:56.724285 20684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:56.716999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37557 (local address 127.0.97.2:40021)
0929 20:57:56.717632 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:56.717653 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:56.717845 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:56.718625 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:56.718630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:56.718665 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:56.719041 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:56.719054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.719993 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.719999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:56.722150 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:56.722167 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.723560 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.723565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:56.723736 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:56.723759 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:57:56.723835 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 20:57:56.723963 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":698,"threads_started":1}
I20260929 20:57:56.725242 20352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0352574ec1047879fbc71812822ee17" candidate_uuid: "c734dce218aa4f47b72ec4ca76bb4894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cebe4b6ba88846e58f074f058bcf768f" is_pre_election: true
I20260929 20:57:56.725787 20352 raft_consensus.cc:2454] T e0352574ec1047879fbc71812822ee17 P cebe4b6ba88846e58f074f058bcf768f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c734dce218aa4f47b72ec4ca76bb4894 in term 0.
I20260929 20:57:56.726436 20416 leader_election.cc:302] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c734dce218aa4f47b72ec4ca76bb4894, cebe4b6ba88846e58f074f058bcf768f; no voters:
I20260929 20:57:56.726758 20681 raft_consensus.cc:2781] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:57:56.726869 20681 raft_consensus.cc:482] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:57:56.726963 20681 raft_consensus.cc:3037] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:56.727818 20671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:56.722380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34971 (local address 127.0.97.3:39255)
0929 20:57:56.722570 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:56.722574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:56.722589 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:57:56.722644 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:56.722647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:56.722701 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:56.722769 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:57:56.723387 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.724040 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.726141 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.727080 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.727174 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:56.727241 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:56.727568 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 20:57:56.727653 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:57:56.727993 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:56.717030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39255 (local address 127.0.97.2:34971)
0929 20:57:56.718814 (+ 1784us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:56.718832 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:57:56.718937 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:57:56.722822 (+ 3885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:57:56.722827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:57:56.722856 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:57:56.723237 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:56.723248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.724172 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.724177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:56.725960 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:57:56.725974 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.727253 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.727259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:57:56.727428 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:56.727461 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:57:56.727559 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 20:57:56.727681 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1664,"thread_start_us":1488,"threads_started":1}
I20260929 20:57:56.728986 20612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0352574ec1047879fbc71812822ee17" candidate_uuid: "c734dce218aa4f47b72ec4ca76bb4894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1980636c3ba841749b26ab959fe40d58" is_pre_election: true
I20260929 20:57:56.729506 20612 raft_consensus.cc:2454] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c734dce218aa4f47b72ec4ca76bb4894 in term 0.
I20260929 20:57:56.729933 20681 raft_consensus.cc:504] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cebe4b6ba88846e58f074f058bcf768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37557 } } peers { permanent_uuid: "1980636c3ba841749b26ab959fe40d58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39255 } } peers { permanent_uuid: "c734dce218aa4f47b72ec4ca76bb4894" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41891 } }
I20260929 20:57:56.730571 20681 leader_election.cc:288] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [CANDIDATE]: Term 1 election: Requested vote from peers cebe4b6ba88846e58f074f058bcf768f (127.0.97.1:37557), 1980636c3ba841749b26ab959fe40d58 (127.0.97.3:39255)
I20260929 20:57:56.730901 20352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0352574ec1047879fbc71812822ee17" candidate_uuid: "c734dce218aa4f47b72ec4ca76bb4894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cebe4b6ba88846e58f074f058bcf768f"
I20260929 20:57:56.730966 20612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0352574ec1047879fbc71812822ee17" candidate_uuid: "c734dce218aa4f47b72ec4ca76bb4894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1980636c3ba841749b26ab959fe40d58"
I20260929 20:57:56.731087 20352 raft_consensus.cc:3037] T e0352574ec1047879fbc71812822ee17 P cebe4b6ba88846e58f074f058bcf768f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:56.731149 20612 raft_consensus.cc:3037] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:56.733248 20612 raft_consensus.cc:2454] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c734dce218aa4f47b72ec4ca76bb4894 in term 1.
I20260929 20:57:56.733264 20352 raft_consensus.cc:2454] T e0352574ec1047879fbc71812822ee17 P cebe4b6ba88846e58f074f058bcf768f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c734dce218aa4f47b72ec4ca76bb4894 in term 1.
I20260929 20:57:56.733613 20416 leader_election.cc:302] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c734dce218aa4f47b72ec4ca76bb4894, cebe4b6ba88846e58f074f058bcf768f; no voters:
I20260929 20:57:56.733896 20681 raft_consensus.cc:2781] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:57:56.734409 20681 raft_consensus.cc:686] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [term 1 LEADER]: Becoming Leader. State: Replica: c734dce218aa4f47b72ec4ca76bb4894, State: Running, Role: LEADER
I20260929 20:57:56.734951 20681 consensus_queue.cc:237] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cebe4b6ba88846e58f074f058bcf768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37557 } } peers { permanent_uuid: "1980636c3ba841749b26ab959fe40d58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39255 } } peers { permanent_uuid: "c734dce218aa4f47b72ec4ca76bb4894" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41891 } }
I20260929 20:57:56.739276 20210 catalog_manager.cc:5495] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 reported cstate change: term changed from 0 to 1, leader changed from <none> to c734dce218aa4f47b72ec4ca76bb4894 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c734dce218aa4f47b72ec4ca76bb4894" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cebe4b6ba88846e58f074f058bcf768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1980636c3ba841749b26ab959fe40d58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c734dce218aa4f47b72ec4ca76bb4894" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41891 } health_report { overall_health: HEALTHY } } }
I20260929 20:57:56.776453 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:56.771206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39400 (local address 127.0.97.2:41891)
0929 20:57:56.771440 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:56.771446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:56.771464 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:57:56.771634 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:56.771638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:56.771725 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:56.771868 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:57:56.772248 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.772957 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.774394 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.774796 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.774907 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:56.775391 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:57:56.775882 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:57:56.776014 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:56.776117 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 20:57:56.776194 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 20:57:56.784951 20612 raft_consensus.cc:1261] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58 [term 1 FOLLOWER]: Refusing update from remote peer c734dce218aa4f47b72ec4ca76bb4894: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:57:56.785162 20352 raft_consensus.cc:1261] T e0352574ec1047879fbc71812822ee17 P cebe4b6ba88846e58f074f058bcf768f [term 1 FOLLOWER]: Refusing update from remote peer c734dce218aa4f47b72ec4ca76bb4894: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:57:56.786033 20686 consensus_queue.cc:1035] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cebe4b6ba88846e58f074f058bcf768f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:57:56.786617 20681 consensus_queue.cc:1035] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980636c3ba841749b26ab959fe40d58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:57:56.804771 20658 tablet.cc:2369] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:57:56.819034 20210 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:57:56.819317 20210 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:57:56.822932 20210 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d208fe7604704669b695a654c45fca42: Sending DeleteTablet for 3 replicas of tablet e0352574ec1047879fbc71812822ee17
I20260929 20:57:56.824096 20592 tablet_service.cc:1514] Processing DeleteTablet for tablet e0352574ec1047879fbc71812822ee17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:57:56 UTC) from {username='slave'} at 127.0.0.1:54144
I20260929 20:57:56.824486 20462 tablet_service.cc:1514] Processing DeleteTablet for tablet e0352574ec1047879fbc71812822ee17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:57:56 UTC) from {username='slave'} at 127.0.0.1:39384
I20260929 20:57:56.824831 20332 tablet_service.cc:1514] Processing DeleteTablet for tablet e0352574ec1047879fbc71812822ee17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:57:56 UTC) from {username='slave'} at 127.0.0.1:39026
I20260929 20:57:56.824994 20699 tablet_replica.cc:330] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58: stopping tablet replica
I20260929 20:57:56.825423 20699 raft_consensus.cc:2229] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:57:56.825947 20699 raft_consensus.cc:2258] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:57:56.826025 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 20270
I20260929 20:57:56.828169 20699 ts_tablet_manager.cc:1905] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:57:56.835417 20699 ts_tablet_manager.cc:1918] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:57:56.835635 20699 log.cc:1196] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-2/wal/wals/e0352574ec1047879fbc71812822ee17
I20260929 20:57:56.836177 20699 ts_tablet_manager.cc:1939] T e0352574ec1047879fbc71812822ee17 P 1980636c3ba841749b26ab959fe40d58: Deleting consensus metadata
I20260929 20:57:56.838204 20195 catalog_manager.cc:4841] TS 1980636c3ba841749b26ab959fe40d58 (127.0.97.3:39255): tablet e0352574ec1047879fbc71812822ee17 (table test-table [id=b246b18217dd4fcc91da2d0ba4e7271c]) successfully deleted
I20260929 20:57:56.843423 20700 tablet_replica.cc:330] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894: stopping tablet replica
I20260929 20:57:56.844015 20700 raft_consensus.cc:2229] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:57:56.844746 20700 raft_consensus.cc:2258] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:57:56.845358 20701 tablet_replica.cc:330] T e0352574ec1047879fbc71812822ee17 P cebe4b6ba88846e58f074f058bcf768f: stopping tablet replica
I20260929 20:57:56.845785 20701 raft_consensus.cc:2229] T e0352574ec1047879fbc71812822ee17 P cebe4b6ba88846e58f074f058bcf768f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:57:56.846314 20701 raft_consensus.cc:2258] T e0352574ec1047879fbc71812822ee17 P cebe4b6ba88846e58f074f058bcf768f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:57:56.849869 20701 ts_tablet_manager.cc:1905] T e0352574ec1047879fbc71812822ee17 P cebe4b6ba88846e58f074f058bcf768f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:57:56.850418 20700 ts_tablet_manager.cc:1905] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:57:56.851217 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:56.827653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39034 (local address 127.0.97.1:37557)
0929 20:57:56.827823 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:56.827829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:56.842752 (+ 14923us) server_negotiation.cc:407] Connection header received
0929 20:57:56.842948 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:56.842954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:56.843105 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:56.843210 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:56.843911 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.845023 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.847208 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.847805 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.847905 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:56.849420 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:56.849432 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:56.849449 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:56.850258 (+ 809us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:56.850325 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:56.850357 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:56.850488 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:56.850702 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 20:57:56.850840 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:57:56.852396 20392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:56.858783 20700 ts_tablet_manager.cc:1918] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:57:56.858970 20700 log.cc:1196] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894: Deleting WAL directory at /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715169435170-388-0/minicluster-data/ts-1/wal/wals/e0352574ec1047879fbc71812822ee17
I20260929 20:57:56.859517 20700 ts_tablet_manager.cc:1939] T e0352574ec1047879fbc71812822ee17 P c734dce218aa4f47b72ec4ca76bb4894: Deleting consensus metadata
I20260929 20:57:56.861474 20197 catalog_manager.cc:4841] TS c734dce218aa4f47b72ec4ca76bb4894 (127.0.97.2:41891): tablet e0352574ec1047879fbc71812822ee17 (table test-table [id=b246b18217dd4fcc91da2d0ba4e7271c]) successfully deleted
I20260929 20:57:56.944111 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 20270
I20260929 20:57:56.965587 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 20400
W20260929 20:57:56.965504 20196 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:37557 (error 108)
I20260929 20:57:56.967906 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:56.967431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37557 (local address 127.0.0.1:39038)
0929 20:57:56.967598 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:56.967726 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 20:57:56.968747 20196 catalog_manager.cc:4584] TS cebe4b6ba88846e58f074f058bcf768f (127.0.97.1:37557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0352574ec1047879fbc71812822ee17: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37557: connect: Connection refused (error 111)
I20260929 20:57:56.973603 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:56.966675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.0.97.2:41891)
0929 20:57:56.966856 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:56.966861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:56.966880 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:57:56.966936 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:56.966940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:56.967016 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:56.967105 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:56.967642 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.968537 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.970482 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:56.970949 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:56.971048 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:56.972235 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:56.972244 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:56.972258 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:56.972876 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:56.972928 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:56.972951 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:56.973040 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:56.973180 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 20:57:56.973260 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:57:56.974346 20522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:57.022418 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:57.021807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37557 (local address 127.0.0.1:39054)
0929 20:57:57.022064 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:57.022205 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 20:57:57.023521 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:57.023142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37557 (local address 127.0.0.1:39060)
0929 20:57:57.023285 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:57.023384 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 20:57:57.084312 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:57.083824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37557 (local address 127.0.0.1:39074)
0929 20:57:57.084058 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:57.084151 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 20:57:57.086239 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:57.085036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37557 (local address 127.0.0.1:39088)
0929 20:57:57.085948 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:57.086053 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":785}
I20260929 20:57:57.086807 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 20400
I20260929 20:57:57.101899 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 20530
I20260929 20:57:57.108654 20671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:57.102635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.0.97.3:39255)
0929 20:57:57.102779 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:57.102783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:57.102848 (+ 65us) server_negotiation.cc:407] Connection header received
0929 20:57:57.102989 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:57.102993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:57.103086 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:57.103173 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:57.103556 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:57.104376 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:57.105685 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:57.106067 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:57.106164 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:57.107344 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:57.107353 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:57.107367 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:57:57.107992 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:57.108047 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:57:57.108066 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:57.108151 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:57.108321 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 20:57:57.108409 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 20:57:57.109295 20652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:57.167128 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:57.166568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37557 (local address 127.0.0.1:39102)
0929 20:57:57.166825 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:57.166947 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 20:57:57.168198 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:57.167768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37557 (local address 127.0.0.1:39118)
0929 20:57:57.167979 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 20:57:57.168072 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 20:57:57.197281 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 20530
I20260929 20:57:57.210637 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 20180
I20260929 20:57:57.211752 20241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:57.319604 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 20180
2026-09-29T20:57:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2623 ms)
[----------] 6 tests from EncryptionPolicyTest (20478 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:57:57 dist-test-slave-m9g1 krb5kdc[20710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:57:57 dist-test-slave-m9g1 krb5kdc[20710](info): set up 2 sockets
Sep 29 20:57:57 dist-test-slave-m9g1 krb5kdc[20710](info): commencing operation
krb5kdc: starting...
W20260929 20:57:58.346288 388 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:57:58 dist-test-slave-m9g1 krb5kdc[20710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:57:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:57:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:57:58.587757 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40565
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:57:58.912930 20723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:57:58.913213 20723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:57:58.913331 20723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:57:58.920481 20723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:57:58.920596 20723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:57:58.920666 20723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:57:58.920747 20723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:57:58.920796 20723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:57:58.931020 20723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40565
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:57:58.932331 20723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:57:58.933653 20723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:57:58.944883 20728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:58.944849 20729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:57:58.944813 20731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:57:58.945649 20723 server_base.cc:1034] running on GCE node
I20260929 20:57:58.946445 20723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:57:58.947818 20723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:57:58.949013 20723 hybrid_clock.cc:648] HybridClock initialized: now 1790715478948966 us; error 44 us; skew 500 ppm
I20260929 20:57:58.949345 20723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:57:58 dist-test-slave-m9g1 krb5kdc[20710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:57:58.952399 20723 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:57:58.953850 20723 webserver.cc:460] Webserver started at http://127.0.0.1:37943/ using document root <none> and password file <none>
I20260929 20:57:58.954607 20723 fs_manager.cc:359] Metadata directory not provided
I20260929 20:57:58.954725 20723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:57:58.955053 20723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:57:58.958102 20723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "78c1c80f642643539380b26329c9ad39"
format_stamp: "Formatted at 2026-09-29 20:57:58 on dist-test-slave-m9g1"
server_key: "440a44b3bffdd1c0fa7adac0cc429f2a"
server_key_iv: "33541c5fbdbbb0d01f7041824362e038"
server_key_version: "encryptionkey@0"
I20260929 20:57:58.958892 20723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "78c1c80f642643539380b26329c9ad39"
format_stamp: "Formatted at 2026-09-29 20:57:58 on dist-test-slave-m9g1"
server_key: "440a44b3bffdd1c0fa7adac0cc429f2a"
server_key_iv: "33541c5fbdbbb0d01f7041824362e038"
server_key_version: "encryptionkey@0"
I20260929 20:57:58.964112 20723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:57:58.967556 20738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:57:58.968951 20723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:57:58.969092 20723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "78c1c80f642643539380b26329c9ad39"
format_stamp: "Formatted at 2026-09-29 20:57:58 on dist-test-slave-m9g1"
server_key: "440a44b3bffdd1c0fa7adac0cc429f2a"
server_key_iv: "33541c5fbdbbb0d01f7041824362e038"
server_key_version: "encryptionkey@0"
I20260929 20:57:58.969257 20723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:57:58.988449 20723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:57:58.992700 20723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:57:58.992951 20723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:57:59.010125 20723 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40565
I20260929 20:57:59.010144 20789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40565 every 8 connection(s)
I20260929 20:57:59.011538 20723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:57:59.015200 20790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:57:59.021324 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 20723
I20260929 20:57:59.021519 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:57:59.022126 388 external_mini_cluster.cc:1428] Setting key 6e206e9995d7fbead050f0eae668b500
I20260929 20:57:59.025815 20790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39: Bootstrap starting.
I20260929 20:57:59.029402 20790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39: Neither blocks nor log segments found. Creating new log.
I20260929 20:57:59.030469 20790 log.cc:824] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39: Log is configured to *not* fsync() on all Append() calls
I20260929 20:57:59.033427 20790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39: No bootstrap required, opened a new log
Sep 29 20:57:59 dist-test-slave-m9g1 krb5kdc[20710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:57:59.042685 20790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c1c80f642643539380b26329c9ad39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40565 } }
I20260929 20:57:59.043084 20790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:57:59.043186 20790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c1c80f642643539380b26329c9ad39, State: Initialized, Role: FOLLOWER
I20260929 20:57:59.043944 20790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c1c80f642643539380b26329c9ad39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40565 } }
I20260929 20:57:59.044255 20790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:57:59.044469 20790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:57:59.044651 20790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:57:59.045564 20793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:59.024106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39903 (local address 127.0.0.1:40565)
0929 20:57:59.024853 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:59.024865 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:59.024906 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:57:59.026692 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:59.026717 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:59.027141 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:59.027489 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:59.029196 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:59.030606 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:59.032222 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:59.032885 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:59.033019 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:59.037655 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:59.037663 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:59.037678 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:59.037715 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:59.040995 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:59.041986 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:59.041994 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:59.042001 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:59.042182 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:59.042674 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:59.042677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:59.042679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:59.043335 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:59.043614 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:59.044091 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 20:57:59.044439 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":340,"threads_started":1}
I20260929 20:57:59.047257 20790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c1c80f642643539380b26329c9ad39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40565 } }
I20260929 20:57:59.047849 20790 leader_election.cc:302] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78c1c80f642643539380b26329c9ad39; no voters:
I20260929 20:57:59.048523 20790 leader_election.cc:288] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:57:59.048620 20795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:57:59.050065 20795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39 [term 1 LEADER]: Becoming Leader. State: Replica: 78c1c80f642643539380b26329c9ad39, State: Running, Role: LEADER
I20260929 20:57:59.050215 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:57:59.050658 20795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c1c80f642643539380b26329c9ad39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40565 } }
I20260929 20:57:59.050798 20790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:57:59.053303 20796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78c1c80f642643539380b26329c9ad39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c1c80f642643539380b26329c9ad39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40565 } } }
I20260929 20:57:59.053272 20797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78c1c80f642643539380b26329c9ad39. Latest consensus state: current_term: 1 leader_uuid: "78c1c80f642643539380b26329c9ad39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c1c80f642643539380b26329c9ad39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40565 } } }
I20260929 20:57:59.053757 20796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39 [sys.catalog]: This master's current role is: LEADER
I20260929 20:57:59.053845 20797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39 [sys.catalog]: This master's current role is: LEADER
I20260929 20:57:59.054644 20803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:57:59.073030 20803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:57:59.085443 20803 catalog_manager.cc:1347] Generated new cluster ID: 0c5a75dd2a2449549befa045d0d31dd8
I20260929 20:57:59.085631 20803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:57:59.087818 20818 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 20:57:59.100433 20793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:57:59.081495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39971 (local address 127.0.0.1:40565)
0929 20:57:59.081692 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:57:59.081699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:57:59.086270 (+ 4571us) server_negotiation.cc:407] Connection header received
0929 20:57:59.087546 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:57:59.087552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:57:59.087653 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:57:59.087736 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:57:59.089392 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:59.090345 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:59.091635 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:57:59.092084 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:57:59.092200 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:57:59.094608 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:57:59.094613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:57:59.094616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:57:59.094649 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:57:59.097312 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:59.098330 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:59.098336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:59.098338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:59.098490 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:57:59.098941 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:57:59.098945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:57:59.098947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:57:59.099227 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:57:59.099473 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 20:57:59.099875 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 20:57:59.100067 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:57:59.113142 20803 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:57:59.114854 20803 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:57:59.121541 20803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78c1c80f642643539380b26329c9ad39: Generated new TSK 0
I20260929 20:57:59.122361 20803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:57:59.133229 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 20723
I20260929 20:57:59.134490 20785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:57:59.226289 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 20723
W20260929 20:57:59.240284 20708 connection.cc:535] client connection to 127.0.0.1:40565 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40565): Connection reset by peer (error 104)
2026-09-29T20:57:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1898 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:57:59 dist-test-slave-m9g1 krb5kdc[20822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:57:59 dist-test-slave-m9g1 krb5kdc[20822](info): set up 2 sockets
Sep 29 20:57:59 dist-test-slave-m9g1 krb5kdc[20822](info): commencing operation
krb5kdc: starting...
W20260929 20:58:00.246129 388 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:58:00 dist-test-slave-m9g1 krb5kdc[20822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:58:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:58:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:58:00.484804 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36569
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:58:00.808871 20835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:00.809168 20835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:00.809260 20835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:00.816588 20835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:00.816711 20835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:00.816762 20835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:00.816824 20835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:00.816874 20835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:00.826880 20835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36569
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:00.828136 20835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:00.829366 20835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:00.841586 20841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:00.841615 20843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:00.841789 20840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:00.842592 20835 server_base.cc:1034] running on GCE node
I20260929 20:58:00.843395 20835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:00.844771 20835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:00.845970 20835 hybrid_clock.cc:648] HybridClock initialized: now 1790715480845918 us; error 43 us; skew 500 ppm
I20260929 20:58:00.846349 20835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:00 dist-test-slave-m9g1 krb5kdc[20822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:00.849385 20835 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:58:00.850880 20835 webserver.cc:460] Webserver started at http://127.0.0.1:39055/ using document root <none> and password file <none>
I20260929 20:58:00.851531 20835 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:00.851635 20835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:00.851943 20835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:00.854916 20835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "dea6d019ff6442b5885577f26f5ad1ce"
format_stamp: "Formatted at 2026-09-29 20:58:00 on dist-test-slave-m9g1"
server_key: "408d220f27d63a84df5c4cd1cdb7b0ce"
server_key_iv: "14dfab7ca3be3408696f68158e030646"
server_key_version: "encryptionkey@0"
I20260929 20:58:00.855623 20835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "dea6d019ff6442b5885577f26f5ad1ce"
format_stamp: "Formatted at 2026-09-29 20:58:00 on dist-test-slave-m9g1"
server_key: "408d220f27d63a84df5c4cd1cdb7b0ce"
server_key_iv: "14dfab7ca3be3408696f68158e030646"
server_key_version: "encryptionkey@0"
I20260929 20:58:00.860788 20835 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:58:00.864476 20850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:00.865942 20835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:58:00.866113 20835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "dea6d019ff6442b5885577f26f5ad1ce"
format_stamp: "Formatted at 2026-09-29 20:58:00 on dist-test-slave-m9g1"
server_key: "408d220f27d63a84df5c4cd1cdb7b0ce"
server_key_iv: "14dfab7ca3be3408696f68158e030646"
server_key_version: "encryptionkey@0"
I20260929 20:58:00.866389 20835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:00.880311 20835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:00.884516 20835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:00.884800 20835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:00.902509 20835 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36569
I20260929 20:58:00.902529 20901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36569 every 8 connection(s)
I20260929 20:58:00.903960 20835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:58:00.907949 20902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:00.907994 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 20835
I20260929 20:58:00.908164 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:58:00.908598 388 external_mini_cluster.cc:1428] Setting key 6aa708250dfc10aef57666fbe79d9ae4
I20260929 20:58:00.921811 20902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce: Bootstrap starting.
I20260929 20:58:00.922325 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:00.910523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56715 (local address 127.0.0.1:36569)
0929 20:58:00.911353 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:00.911365 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:00.911407 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:58:00.912991 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:00.913015 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:00.913451 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:00.913802 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:00.914467 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:00.915829 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:00.917637 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:00.918420 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:00.918551 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:00.919841 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:00.919848 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:00.919864 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:00.920444 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:00.920526 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:00.920555 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:00.920670 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:00.920996 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 20:58:00.921199 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":370,"threads_started":1}
I20260929 20:58:00.925405 20902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:00.926424 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:58:00.926553 20902 log.cc:824] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:00.929452 20902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce: No bootstrap required, opened a new log
I20260929 20:58:00.937675 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:00.930911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59835 (local address 127.0.0.1:36569)
0929 20:58:00.931109 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:00.931115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:00.931515 (+ 400us) server_negotiation.cc:407] Connection header received
0929 20:58:00.931668 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:00.931674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:00.931763 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:00.931880 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:00.932334 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:00.933461 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:00.934985 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:00.935687 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:00.935814 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:00.936953 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:00.936959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:00.936964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:00.936985 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:00.937045 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:00.937065 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:00.937177 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:00.937294 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 20:58:00.937354 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:58:00.938623 20902 raft_consensus.cc:348] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dea6d019ff6442b5885577f26f5ad1ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36569 } }
I20260929 20:58:00.939122 20902 raft_consensus.cc:374] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:58:00.939231 20902 raft_consensus.cc:729] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea6d019ff6442b5885577f26f5ad1ce, State: Initialized, Role: FOLLOWER
I20260929 20:58:00.939791 20902 consensus_queue.cc:260] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dea6d019ff6442b5885577f26f5ad1ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36569 } }
I20260929 20:58:00.940006 20902 raft_consensus.cc:388] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:58:00.940104 20902 raft_consensus.cc:482] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:58:00.940215 20902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:00.942658 20902 raft_consensus.cc:504] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dea6d019ff6442b5885577f26f5ad1ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36569 } }
I20260929 20:58:00.943147 20902 leader_election.cc:302] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dea6d019ff6442b5885577f26f5ad1ce; no voters:
I20260929 20:58:00.943641 20902 leader_election.cc:288] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:58:00.943809 20912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:58:00.944450 20912 raft_consensus.cc:686] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce [term 1 LEADER]: Becoming Leader. State: Replica: dea6d019ff6442b5885577f26f5ad1ce, State: Running, Role: LEADER
I20260929 20:58:00.945007 20912 consensus_queue.cc:237] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dea6d019ff6442b5885577f26f5ad1ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36569 } }
I20260929 20:58:00.945587 20902 sys_catalog.cc:564] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:00.947683 20913 sys_catalog.cc:455] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dea6d019ff6442b5885577f26f5ad1ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dea6d019ff6442b5885577f26f5ad1ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36569 } } }
I20260929 20:58:00.947818 20914 sys_catalog.cc:455] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader dea6d019ff6442b5885577f26f5ad1ce. Latest consensus state: current_term: 1 leader_uuid: "dea6d019ff6442b5885577f26f5ad1ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dea6d019ff6442b5885577f26f5ad1ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36569 } } }
I20260929 20:58:00.948205 20914 sys_catalog.cc:458] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:00.948817 20913 sys_catalog.cc:458] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:00.950309 20921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:58:00.954473 20921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:58:00.963186 20921 catalog_manager.cc:1347] Generated new cluster ID: 0d70f5713c1543e5851f89f7cddceb3d
I20260929 20:58:00.963316 20921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:58:00.989638 20921 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:58:00.991794 20921 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:58:01.002377 20921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dea6d019ff6442b5885577f26f5ad1ce: Generated new TSK 0
I20260929 20:58:01.003551 20921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:58:01.012140 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 20835
W20260929 20:58:01.013801 20897 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56715
W20260929 20:58:01.014681 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 20835: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:58:01.014906 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 20835
W20260929 20:58:01.029057 20820 connection.cc:535] client connection to 127.0.0.1:36569 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36569): Connection reset by peer (error 104)
2026-09-29T20:58:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1788 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:58:01 dist-test-slave-m9g1 krb5kdc[20933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:58:01 dist-test-slave-m9g1 krb5kdc[20933](info): set up 2 sockets
Sep 29 20:58:01 dist-test-slave-m9g1 krb5kdc[20933](info): commencing operation
krb5kdc: starting...
W20260929 20:58:02.022101 388 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.013s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:58:02 dist-test-slave-m9g1 krb5kdc[20933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:58:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:58:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:58:02.258940 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45735
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:58:02.582401 20946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:02.582710 20946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:02.582854 20946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:02.590081 20946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:02.590255 20946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:02.590318 20946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:02.590380 20946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:02.590422 20946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:02.600242 20946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45735
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:02.601497 20946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:02.602771 20946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:02.614166 20951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:02.614219 20954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:02.614259 20952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:02.615360 20946 server_base.cc:1034] running on GCE node
I20260929 20:58:02.616104 20946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:02.617507 20946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:02.618709 20946 hybrid_clock.cc:648] HybridClock initialized: now 1790715482618654 us; error 46 us; skew 500 ppm
I20260929 20:58:02.619051 20946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:02 dist-test-slave-m9g1 krb5kdc[20933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:02.629237 20946 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:58:02.630980 20946 webserver.cc:460] Webserver started at http://127.0.0.1:37013/ using document root <none> and password file <none>
I20260929 20:58:02.631676 20946 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:02.631816 20946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:02.632108 20946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:02.635118 20946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "90424bc74c764f049b13088cfb9f88f5"
format_stamp: "Formatted at 2026-09-29 20:58:02 on dist-test-slave-m9g1"
server_key: "d95292ca962ca5658976a793f643d032"
server_key_iv: "6b8a3cf1d6334148daa7733afe216d51"
server_key_version: "encryptionkey@0"
I20260929 20:58:02.635862 20946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "90424bc74c764f049b13088cfb9f88f5"
format_stamp: "Formatted at 2026-09-29 20:58:02 on dist-test-slave-m9g1"
server_key: "d95292ca962ca5658976a793f643d032"
server_key_iv: "6b8a3cf1d6334148daa7733afe216d51"
server_key_version: "encryptionkey@0"
I20260929 20:58:02.640852 20946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:58:02.644160 20961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:02.645502 20946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:58:02.645659 20946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "90424bc74c764f049b13088cfb9f88f5"
format_stamp: "Formatted at 2026-09-29 20:58:02 on dist-test-slave-m9g1"
server_key: "d95292ca962ca5658976a793f643d032"
server_key_iv: "6b8a3cf1d6334148daa7733afe216d51"
server_key_version: "encryptionkey@0"
I20260929 20:58:02.645859 20946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:02.651332 20946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:02.655251 20946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:02.655520 20946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:02.673213 20946 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45735
I20260929 20:58:02.673247 21012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45735 every 8 connection(s)
I20260929 20:58:02.674680 20946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:58:02.678458 21013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:02.682008 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 20946
I20260929 20:58:02.682161 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:58:02.682585 388 external_mini_cluster.cc:1428] Setting key f378b8e0bc068f4fa35c8db9dc69fa18
I20260929 20:58:02.690105 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:02.684370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36479 (local address 127.0.0.1:45735)
0929 20:58:02.685119 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:02.685130 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:02.685172 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:58:02.686848 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:02.686874 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:02.687253 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:02.687632 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:02.688598 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:02.688607 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:02.688620 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:02.688635 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:02.688691 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:02.688710 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:02.688914 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:02.689046 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 20:58:02.689215 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":325,"threads_started":1}
I20260929 20:58:02.690403 21013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5: Bootstrap starting.
I20260929 20:58:02.693522 21013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:02.693580 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:58:02.694547 21013 log.cc:824] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:02.697288 21013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5: No bootstrap required, opened a new log
I20260929 20:58:02.700066 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:02.697595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52219 (local address 127.0.0.1:45735)
0929 20:58:02.697803 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:02.697810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:02.698175 (+ 365us) server_negotiation.cc:407] Connection header received
0929 20:58:02.698382 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:02.698386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:02.698432 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:02.698507 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:02.699426 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:02.699430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:02.699433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:02.699448 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:02.699490 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:02.699505 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:02.699601 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:02.699709 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 20:58:02.699761 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:58:02.705601 21013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90424bc74c764f049b13088cfb9f88f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45735 } }
I20260929 20:58:02.705979 21013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:58:02.706065 21013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90424bc74c764f049b13088cfb9f88f5, State: Initialized, Role: FOLLOWER
I20260929 20:58:02.706622 21013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90424bc74c764f049b13088cfb9f88f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45735 } }
I20260929 20:58:02.706842 21013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:58:02.706969 21013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:58:02.707118 21013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:02.709275 21013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90424bc74c764f049b13088cfb9f88f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45735 } }
I20260929 20:58:02.709769 21013 leader_election.cc:302] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90424bc74c764f049b13088cfb9f88f5; no voters:
I20260929 20:58:02.710407 21013 leader_election.cc:288] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:58:02.710528 21023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:58:02.711064 21023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5 [term 1 LEADER]: Becoming Leader. State: Replica: 90424bc74c764f049b13088cfb9f88f5, State: Running, Role: LEADER
I20260929 20:58:02.711704 21023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90424bc74c764f049b13088cfb9f88f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45735 } }
I20260929 20:58:02.712944 21013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:02.714325 21025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90424bc74c764f049b13088cfb9f88f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90424bc74c764f049b13088cfb9f88f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45735 } } }
I20260929 20:58:02.714671 21024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90424bc74c764f049b13088cfb9f88f5. Latest consensus state: current_term: 1 leader_uuid: "90424bc74c764f049b13088cfb9f88f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90424bc74c764f049b13088cfb9f88f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45735 } } }
I20260929 20:58:02.714846 21025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5 [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:02.714910 21024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5 [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:02.716538 21029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:58:02.721329 21029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:58:02.731058 21029 catalog_manager.cc:1347] Generated new cluster ID: 42f4dbda57b94120a8953779eece8093
I20260929 20:58:02.731166 21029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:58:02.758639 21029 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:58:02.759168 21029 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:58:02.770057 21029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90424bc74c764f049b13088cfb9f88f5: Generated new TSK 0
I20260929 20:58:02.771234 21029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:58:02.785274 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 20946
W20260929 20:58:02.786362 21008 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36479
W20260929 20:58:02.787071 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 20946: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:58:02.787242 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 20946
2026-09-29T20:58:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1772 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:58:02 dist-test-slave-m9g1 krb5kdc[21044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:58:02 dist-test-slave-m9g1 krb5kdc[21044](info): set up 2 sockets
Sep 29 20:58:02 dist-test-slave-m9g1 krb5kdc[21044](info): commencing operation
krb5kdc: starting...
W20260929 20:58:03.810384 388 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:58:03 dist-test-slave-m9g1 krb5kdc[21044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:58:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:58:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:58:04.049600 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:58:04.368017 21057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:04.368295 21057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:04.368429 21057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:04.375667 21057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:04.375792 21057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:04.375847 21057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:04.375928 21057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:04.375988 21057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:04.385850 21057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:04.387143 21057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:04.388412 21057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:04.399681 21062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:04.399734 21063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:04.399755 21065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:04.400470 21057 server_base.cc:1034] running on GCE node
I20260929 20:58:04.401196 21057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:04.402668 21057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:04.403877 21057 hybrid_clock.cc:648] HybridClock initialized: now 1790715484403827 us; error 43 us; skew 500 ppm
I20260929 20:58:04.404217 21057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:04 dist-test-slave-m9g1 krb5kdc[21044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:04.407292 21057 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:58:04.408756 21057 webserver.cc:460] Webserver started at http://127.0.0.1:37029/ using document root <none> and password file <none>
I20260929 20:58:04.409407 21057 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:04.409505 21057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:04.409765 21057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:04.412791 21057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "8115dff86ce84888b2d9c7a2d0b41471"
format_stamp: "Formatted at 2026-09-29 20:58:04 on dist-test-slave-m9g1"
server_key: "0c319fa592ac82f7eadc97fd30999e54"
server_key_iv: "d362905789ee8142471c10768ca20faa"
server_key_version: "encryptionkey@0"
I20260929 20:58:04.413479 21057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "8115dff86ce84888b2d9c7a2d0b41471"
format_stamp: "Formatted at 2026-09-29 20:58:04 on dist-test-slave-m9g1"
server_key: "0c319fa592ac82f7eadc97fd30999e54"
server_key_iv: "d362905789ee8142471c10768ca20faa"
server_key_version: "encryptionkey@0"
I20260929 20:58:04.418473 21057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 20:58:04.421872 21072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:04.423245 21057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:58:04.423393 21057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "8115dff86ce84888b2d9c7a2d0b41471"
format_stamp: "Formatted at 2026-09-29 20:58:04 on dist-test-slave-m9g1"
server_key: "0c319fa592ac82f7eadc97fd30999e54"
server_key_iv: "d362905789ee8142471c10768ca20faa"
server_key_version: "encryptionkey@0"
I20260929 20:58:04.423574 21057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:04.452545 21057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:04.456511 21057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:04.456755 21057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:04.473621 21057 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44539
I20260929 20:58:04.473652 21123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44539 every 8 connection(s)
I20260929 20:58:04.475013 21057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:58:04.478873 21124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:04.483122 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 21057
I20260929 20:58:04.483321 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:58:04.483729 388 external_mini_cluster.cc:1428] Setting key 261bb58fb886a8ddc0f6bdd71ab3b47e
I20260929 20:58:04.490371 21124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471: Bootstrap starting.
I20260929 20:58:04.493933 21124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:04.495066 21124 log.cc:824] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:04.498538 21124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471: No bootstrap required, opened a new log
Sep 29 20:58:04 dist-test-slave-m9g1 krb5kdc[21044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:58:04.506129 21124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8115dff86ce84888b2d9c7a2d0b41471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44539 } }
I20260929 20:58:04.506529 21124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:58:04.506625 21124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8115dff86ce84888b2d9c7a2d0b41471, State: Initialized, Role: FOLLOWER
I20260929 20:58:04.507174 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:04.485448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52713 (local address 127.0.0.1:44539)
0929 20:58:04.486321 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:04.486334 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:04.486376 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:58:04.488565 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:04.488589 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:04.489030 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:04.489349 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:04.491137 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:04.492498 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:04.494266 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:04.494957 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:04.495092 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:04.499825 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:04.499833 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:04.499848 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:04.499891 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:04.503016 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:04.503689 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:04.503696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:04.503702 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:04.503882 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:04.504258 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:04.504261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:04.504263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:04.504842 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:04.505126 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:04.505565 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 20:58:04.505973 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":405,"threads_started":1}
I20260929 20:58:04.507225 21124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8115dff86ce84888b2d9c7a2d0b41471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44539 } }
I20260929 20:58:04.507503 21124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:58:04.507634 21124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:58:04.507798 21124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:04.510179 21124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8115dff86ce84888b2d9c7a2d0b41471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44539 } }
I20260929 20:58:04.510775 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:58:04.510778 21124 leader_election.cc:302] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8115dff86ce84888b2d9c7a2d0b41471; no voters:
I20260929 20:58:04.511384 21124 leader_election.cc:288] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:58:04.511497 21129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:58:04.513634 21124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:04.515008 21129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471 [term 1 LEADER]: Becoming Leader. State: Replica: 8115dff86ce84888b2d9c7a2d0b41471, State: Running, Role: LEADER
I20260929 20:58:04.515686 21129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8115dff86ce84888b2d9c7a2d0b41471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44539 } }
I20260929 20:58:04.518994 21130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8115dff86ce84888b2d9c7a2d0b41471" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8115dff86ce84888b2d9c7a2d0b41471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44539 } } }
I20260929 20:58:04.519280 21130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471 [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:04.518428 21131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8115dff86ce84888b2d9c7a2d0b41471. Latest consensus state: current_term: 1 leader_uuid: "8115dff86ce84888b2d9c7a2d0b41471" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8115dff86ce84888b2d9c7a2d0b41471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44539 } } }
I20260929 20:58:04.519662 21131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471 [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:04.529599 21140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:58:04.531385 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:04.515538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58099 (local address 127.0.0.1:44539)
0929 20:58:04.515689 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:04.515695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:04.515770 (+ 75us) server_negotiation.cc:407] Connection header received
0929 20:58:04.516013 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:04.516019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:04.516079 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:04.516129 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:04.517751 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:04.518849 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:04.520686 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:04.521429 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:04.521546 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:04.524954 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:04.524961 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:04.524967 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:04.525009 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:04.528372 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:04.528987 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:04.528993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:04.528996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:04.529180 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:04.529669 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:04.529676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:04.529679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:04.530098 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:04.530391 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:04.530725 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 20:58:04.530967 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:58:04.535421 21140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:58:04.543435 21140 catalog_manager.cc:1347] Generated new cluster ID: d61cd732a89b4a128b683aec6232a91f
I20260929 20:58:04.543545 21140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:58:04.558079 21140 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:58:04.560053 21140 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:58:04.568323 21140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8115dff86ce84888b2d9c7a2d0b41471: Generated new TSK 0
I20260929 20:58:04.569439 21140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:58:04.583709 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 21057
I20260929 20:58:04.585484 21119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:58:04.675407 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 21057
2026-09-29T20:58:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1888 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:58:04 dist-test-slave-m9g1 krb5kdc[21157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:58:04 dist-test-slave-m9g1 krb5kdc[21157](info): set up 2 sockets
Sep 29 20:58:04 dist-test-slave-m9g1 krb5kdc[21157](info): commencing operation
krb5kdc: starting...
W20260929 20:58:05.686376 388 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:58:05 dist-test-slave-m9g1 krb5kdc[21157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:58:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:58:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:58:05.927413 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39107
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:58:06.252051 21170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:06.252369 21170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:06.252506 21170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:06.259771 21170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:06.259900 21170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:06.259980 21170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:06.260058 21170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:06.260118 21170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:06.269840 21170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:06.271135 21170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:06.272477 21170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:06.283833 21178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:06.283877 21176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:06.283932 21175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:06.284621 21170 server_base.cc:1034] running on GCE node
I20260929 20:58:06.285468 21170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:06.287019 21170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:06.288224 21170 hybrid_clock.cc:648] HybridClock initialized: now 1790715486288171 us; error 50 us; skew 500 ppm
I20260929 20:58:06.288573 21170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:06 dist-test-slave-m9g1 krb5kdc[21157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:06.291759 21170 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:58:06.293144 21170 webserver.cc:460] Webserver started at http://127.0.0.1:43665/ using document root <none> and password file <none>
I20260929 20:58:06.293792 21170 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:06.293888 21170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:06.294133 21170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:06.297170 21170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "c680b321b4f64b0bb651a3530d9135d1"
format_stamp: "Formatted at 2026-09-29 20:58:06 on dist-test-slave-m9g1"
server_key: "c6c8107fdf527da85330f263c236a7ee"
server_key_iv: "121a4d19375c4dd6a11cb8d6091b83fa"
server_key_version: "encryptionkey@0"
I20260929 20:58:06.297884 21170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "c680b321b4f64b0bb651a3530d9135d1"
format_stamp: "Formatted at 2026-09-29 20:58:06 on dist-test-slave-m9g1"
server_key: "c6c8107fdf527da85330f263c236a7ee"
server_key_iv: "121a4d19375c4dd6a11cb8d6091b83fa"
server_key_version: "encryptionkey@0"
I20260929 20:58:06.303033 21170 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 20:58:06.306432 21185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:06.307754 21170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:58:06.307904 21170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "c680b321b4f64b0bb651a3530d9135d1"
format_stamp: "Formatted at 2026-09-29 20:58:06 on dist-test-slave-m9g1"
server_key: "c6c8107fdf527da85330f263c236a7ee"
server_key_iv: "121a4d19375c4dd6a11cb8d6091b83fa"
server_key_version: "encryptionkey@0"
I20260929 20:58:06.308082 21170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:06.335619 21170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:06.339531 21170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:06.339812 21170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:06.356463 21170 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39107
I20260929 20:58:06.356491 21236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39107 every 8 connection(s)
I20260929 20:58:06.357731 21170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:58:06.360702 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 21170
I20260929 20:58:06.360898 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:58:06.361313 388 external_mini_cluster.cc:1428] Setting key ece23a55f5785782791ad849e81c8dc4
I20260929 20:58:06.361362 21237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:06.375703 21240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:06.363057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55239 (local address 127.0.0.1:39107)
0929 20:58:06.363849 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:06.363863 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:06.363917 (+ 54us) server_negotiation.cc:407] Connection header received
0929 20:58:06.365974 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:06.366000 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:06.366523 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:06.366959 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:06.367472 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:06.369422 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:06.370825 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:06.371849 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:06.372047 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:06.373028 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:06.373041 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:06.373064 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:06.373772 (+ 708us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:06.373875 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:06.373904 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:06.374015 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:06.374177 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 20:58:06.374457 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":295,"threads_started":1}
I20260929 20:58:06.375927 21237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1: Bootstrap starting.
I20260929 20:58:06.379031 21237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:06.379101 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:58:06.379961 21237 log.cc:824] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:06.382819 21237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1: No bootstrap required, opened a new log
I20260929 20:58:06.389078 21240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:06.383300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59281 (local address 127.0.0.1:39107)
0929 20:58:06.383438 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:06.383442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:06.383884 (+ 442us) server_negotiation.cc:407] Connection header received
0929 20:58:06.384050 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:06.384054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:06.384112 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:06.384191 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:06.384667 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:06.385422 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:06.386792 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:06.387292 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:06.387404 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:06.388418 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:06.388422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:06.388425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:06.388442 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:06.388481 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:06.388495 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:06.388568 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:06.388694 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 20:58:06.388795 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 20:58:06.390888 21237 raft_consensus.cc:348] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c680b321b4f64b0bb651a3530d9135d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39107 } }
I20260929 20:58:06.391260 21237 raft_consensus.cc:374] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:58:06.391356 21237 raft_consensus.cc:729] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c680b321b4f64b0bb651a3530d9135d1, State: Initialized, Role: FOLLOWER
I20260929 20:58:06.391924 21237 consensus_queue.cc:260] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c680b321b4f64b0bb651a3530d9135d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39107 } }
I20260929 20:58:06.392136 21237 raft_consensus.cc:388] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:58:06.392271 21237 raft_consensus.cc:482] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:58:06.392412 21237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:06.394578 21237 raft_consensus.cc:504] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c680b321b4f64b0bb651a3530d9135d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39107 } }
I20260929 20:58:06.395079 21237 leader_election.cc:302] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c680b321b4f64b0bb651a3530d9135d1; no voters:
I20260929 20:58:06.395685 21237 leader_election.cc:288] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:58:06.395795 21247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:58:06.396430 21247 raft_consensus.cc:686] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1 [term 1 LEADER]: Becoming Leader. State: Replica: c680b321b4f64b0bb651a3530d9135d1, State: Running, Role: LEADER
I20260929 20:58:06.396935 21247 consensus_queue.cc:237] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c680b321b4f64b0bb651a3530d9135d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39107 } }
I20260929 20:58:06.397648 21237 sys_catalog.cc:564] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:06.399537 21248 sys_catalog.cc:455] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c680b321b4f64b0bb651a3530d9135d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c680b321b4f64b0bb651a3530d9135d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39107 } } }
I20260929 20:58:06.400242 21248 sys_catalog.cc:458] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1 [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:06.400048 21249 sys_catalog.cc:455] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c680b321b4f64b0bb651a3530d9135d1. Latest consensus state: current_term: 1 leader_uuid: "c680b321b4f64b0bb651a3530d9135d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c680b321b4f64b0bb651a3530d9135d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39107 } } }
I20260929 20:58:06.401036 21249 sys_catalog.cc:458] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1 [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:06.402756 21256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:58:06.408144 21256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:58:06.417665 21256 catalog_manager.cc:1347] Generated new cluster ID: 9b863d00c41c4989a4f44c41d34d39e0
I20260929 20:58:06.417788 21256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:58:06.437546 21256 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:58:06.438854 21256 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:58:06.448820 21256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c680b321b4f64b0bb651a3530d9135d1: Generated new TSK 0
I20260929 20:58:06.449646 21256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:58:06.458001 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 21170
W20260929 20:58:06.459116 21232 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55239
W20260929 20:58:06.459810 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 21170: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:58:06.459940 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 21170
2026-09-29T20:58:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1784 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:58:06 dist-test-slave-m9g1 krb5kdc[21268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:58:06 dist-test-slave-m9g1 krb5kdc[21268](info): set up 2 sockets
Sep 29 20:58:06 dist-test-slave-m9g1 krb5kdc[21268](info): commencing operation
krb5kdc: starting...
W20260929 20:58:07.458403 388 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:58:07 dist-test-slave-m9g1 krb5kdc[21268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:58:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:58:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:58:07.697366 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34239
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:58:08.026870 21281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:08.027170 21281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:08.027295 21281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:08.034723 21281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:08.034883 21281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:08.034999 21281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:08.035109 21281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:08.035197 21281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:08.045353 21281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:08.046782 21281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:08.048152 21281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:08.060523 21289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:08.060599 21286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:08.060534 21287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:08.061275 21281 server_base.cc:1034] running on GCE node
I20260929 20:58:08.062161 21281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:08.063736 21281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:08.064968 21281 hybrid_clock.cc:648] HybridClock initialized: now 1790715488064912 us; error 49 us; skew 500 ppm
I20260929 20:58:08.065343 21281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:08 dist-test-slave-m9g1 krb5kdc[21268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:08.072171 21281 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:58:08.073873 21281 webserver.cc:460] Webserver started at http://127.0.0.1:37257/ using document root <none> and password file <none>
I20260929 20:58:08.074677 21281 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:08.074805 21281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:08.075099 21281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:08.078296 21281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "009a765b1ff541efb69ed27cbcd99a14"
format_stamp: "Formatted at 2026-09-29 20:58:08 on dist-test-slave-m9g1"
server_key: "9304fb377a07069c273a41b51f58d64a"
server_key_iv: "aab242173da81b013ea4eca80396baaf"
server_key_version: "encryptionkey@0"
I20260929 20:58:08.079075 21281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "009a765b1ff541efb69ed27cbcd99a14"
format_stamp: "Formatted at 2026-09-29 20:58:08 on dist-test-slave-m9g1"
server_key: "9304fb377a07069c273a41b51f58d64a"
server_key_iv: "aab242173da81b013ea4eca80396baaf"
server_key_version: "encryptionkey@0"
I20260929 20:58:08.084528 21281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260929 20:58:08.088299 21296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:08.089820 21281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:58:08.090011 21281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "009a765b1ff541efb69ed27cbcd99a14"
format_stamp: "Formatted at 2026-09-29 20:58:08 on dist-test-slave-m9g1"
server_key: "9304fb377a07069c273a41b51f58d64a"
server_key_iv: "aab242173da81b013ea4eca80396baaf"
server_key_version: "encryptionkey@0"
I20260929 20:58:08.090247 21281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:08.096336 21281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:08.100616 21281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:08.100910 21281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:08.118067 21281 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34239
I20260929 20:58:08.118103 21347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34239 every 8 connection(s)
I20260929 20:58:08.119473 21281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:58:08.120721 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 21281
I20260929 20:58:08.120944 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:58:08.121434 388 external_mini_cluster.cc:1428] Setting key b92ed11d502d2cb60d106b9f3572fc60
I20260929 20:58:08.123945 21348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:08.128856 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:08.123206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38603 (local address 127.0.0.1:34239)
0929 20:58:08.123946 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:08.123958 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:08.123999 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:58:08.125617 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:08.125638 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:08.125974 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:08.126354 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:08.127322 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:08.127328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:08.127341 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:08.127354 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:08.127425 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:08.127447 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:08.127602 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:08.127743 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 20:58:08.127920 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":326,"threads_started":1}
I20260929 20:58:08.132664 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:58:08.137085 21348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14: Bootstrap starting.
I20260929 20:58:08.140390 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:08.137966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36739 (local address 127.0.0.1:34239)
0929 20:58:08.138145 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:08.138150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:08.138431 (+ 281us) server_negotiation.cc:407] Connection header received
0929 20:58:08.138616 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:08.138620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:08.138703 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:08.138786 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:08.139729 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:08.139733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:08.139737 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:08.139751 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:08.139790 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:08.139805 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:08.139891 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:08.140066 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 20:58:08.140116 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:58:08.140938 21348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:08.142030 21348 log.cc:824] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:08.144856 21348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14: No bootstrap required, opened a new log
I20260929 20:58:08.152259 21348 raft_consensus.cc:348] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009a765b1ff541efb69ed27cbcd99a14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34239 } }
I20260929 20:58:08.152606 21348 raft_consensus.cc:374] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:58:08.152688 21348 raft_consensus.cc:729] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009a765b1ff541efb69ed27cbcd99a14, State: Initialized, Role: FOLLOWER
I20260929 20:58:08.153172 21348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009a765b1ff541efb69ed27cbcd99a14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34239 } }
I20260929 20:58:08.153364 21348 raft_consensus.cc:388] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:58:08.153462 21348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:58:08.153581 21348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:08.155712 21348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009a765b1ff541efb69ed27cbcd99a14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34239 } }
I20260929 20:58:08.156167 21348 leader_election.cc:302] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 009a765b1ff541efb69ed27cbcd99a14; no voters:
I20260929 20:58:08.156737 21348 leader_election.cc:288] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:58:08.156852 21358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:58:08.157518 21358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14 [term 1 LEADER]: Becoming Leader. State: Replica: 009a765b1ff541efb69ed27cbcd99a14, State: Running, Role: LEADER
I20260929 20:58:08.157996 21358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009a765b1ff541efb69ed27cbcd99a14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34239 } }
I20260929 20:58:08.158797 21348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:08.160760 21359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "009a765b1ff541efb69ed27cbcd99a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009a765b1ff541efb69ed27cbcd99a14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34239 } } }
I20260929 20:58:08.160789 21360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 009a765b1ff541efb69ed27cbcd99a14. Latest consensus state: current_term: 1 leader_uuid: "009a765b1ff541efb69ed27cbcd99a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009a765b1ff541efb69ed27cbcd99a14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34239 } } }
I20260929 20:58:08.161278 21360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14 [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:08.161922 21359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14 [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:08.163990 21363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:58:08.169265 21363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:58:08.178781 21363 catalog_manager.cc:1347] Generated new cluster ID: 5f313805420047838b472f847588eb95
I20260929 20:58:08.178905 21363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:58:08.192157 21363 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:58:08.192598 21363 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:58:08.203145 21363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 009a765b1ff541efb69ed27cbcd99a14: Generated new TSK 0
I20260929 20:58:08.204075 21363 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:58:08.211740 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 21281
W20260929 20:58:08.212872 21343 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38603
W20260929 20:58:08.213634 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 21281: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:58:08.213796 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 21281
2026-09-29T20:58:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1753 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:58:08 dist-test-slave-m9g1 krb5kdc[21379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:58:08 dist-test-slave-m9g1 krb5kdc[21379](info): set up 2 sockets
Sep 29 20:58:08 dist-test-slave-m9g1 krb5kdc[21379](info): commencing operation
krb5kdc: starting...
W20260929 20:58:09.234321 388 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:58:09 dist-test-slave-m9g1 krb5kdc[21379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:58:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:58:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:58:09.472780 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40389
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:58:09.792528 21392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:09.792865 21392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:09.793004 21392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:09.800319 21392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:09.800459 21392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:09.800536 21392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:09.800624 21392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:09.800684 21392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:09.810683 21392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:09.812021 21392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:09.813297 21392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:09.824398 21397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:09.824416 21398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:09.824355 21400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:09.825181 21392 server_base.cc:1034] running on GCE node
I20260929 20:58:09.825896 21392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:09.827317 21392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:09.828532 21392 hybrid_clock.cc:648] HybridClock initialized: now 1790715489828486 us; error 28 us; skew 500 ppm
I20260929 20:58:09.828866 21392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:09 dist-test-slave-m9g1 krb5kdc[21379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:09.831947 21392 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:58:09.833411 21392 webserver.cc:460] Webserver started at http://127.0.0.1:38233/ using document root <none> and password file <none>
I20260929 20:58:09.834096 21392 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:09.834259 21392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:09.834556 21392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:09.837474 21392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "a7134106c43844d0a617e47d26d12b9f"
format_stamp: "Formatted at 2026-09-29 20:58:09 on dist-test-slave-m9g1"
server_key: "454b52b7c7e114b32e6b29b0eceb10f9"
server_key_iv: "b19d4975fce37ed850252ec2d748fde5"
server_key_version: "encryptionkey@0"
I20260929 20:58:09.838238 21392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "a7134106c43844d0a617e47d26d12b9f"
format_stamp: "Formatted at 2026-09-29 20:58:09 on dist-test-slave-m9g1"
server_key: "454b52b7c7e114b32e6b29b0eceb10f9"
server_key_iv: "b19d4975fce37ed850252ec2d748fde5"
server_key_version: "encryptionkey@0"
I20260929 20:58:09.843261 21392 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 20:58:09.846661 21407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:09.847934 21392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:58:09.848091 21392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "a7134106c43844d0a617e47d26d12b9f"
format_stamp: "Formatted at 2026-09-29 20:58:09 on dist-test-slave-m9g1"
server_key: "454b52b7c7e114b32e6b29b0eceb10f9"
server_key_iv: "b19d4975fce37ed850252ec2d748fde5"
server_key_version: "encryptionkey@0"
I20260929 20:58:09.848285 21392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:09.883255 21392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:09.887588 21392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:09.887862 21392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:09.905503 21392 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40389
I20260929 20:58:09.905522 21458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40389 every 8 connection(s)
I20260929 20:58:09.906917 21392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:58:09.910676 21459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:09.915740 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 21392
I20260929 20:58:09.915894 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:58:09.916280 388 external_mini_cluster.cc:1428] Setting key 6f61789dedcb3e990441039ac6c13ad3
I20260929 20:58:09.923017 21459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f: Bootstrap starting.
I20260929 20:58:09.926759 21459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:09.927879 21459 log.cc:824] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:09.930831 21459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f: No bootstrap required, opened a new log
Sep 29 20:58:09 dist-test-slave-m9g1 krb5kdc[21379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:58:09.939742 21459 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7134106c43844d0a617e47d26d12b9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40389 } }
I20260929 20:58:09.940135 21459 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:58:09.940220 21459 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7134106c43844d0a617e47d26d12b9f, State: Initialized, Role: FOLLOWER
I20260929 20:58:09.940876 21459 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7134106c43844d0a617e47d26d12b9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40389 } }
I20260929 20:58:09.941149 21459 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:58:09.941298 21459 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:58:09.941484 21459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:09.944242 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:09.918225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:57407 (local address 127.0.0.1:40389)
0929 20:58:09.919076 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:09.919096 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:09.919149 (+ 53us) server_negotiation.cc:407] Connection header received
0929 20:58:09.921624 (+ 2475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:09.921655 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:09.922228 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:09.922698 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:09.924045 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:09.926017 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:09.927348 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:09.928357 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:09.928560 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:09.935618 (+ 7058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:09.935629 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:09.935649 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:09.935705 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:09.939514 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:09.940382 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:09.940390 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:09.940398 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:09.940617 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:09.941074 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:09.941078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:09.941081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:09.941823 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:09.942157 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:09.942559 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 20:58:09.942921 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":312,"threads_started":1}
I20260929 20:58:09.944715 21459 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7134106c43844d0a617e47d26d12b9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40389 } }
I20260929 20:58:09.945232 21459 leader_election.cc:302] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7134106c43844d0a617e47d26d12b9f; no voters:
I20260929 20:58:09.945879 21459 leader_election.cc:288] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:58:09.946035 21464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:58:09.947327 21464 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f [term 1 LEADER]: Becoming Leader. State: Replica: a7134106c43844d0a617e47d26d12b9f, State: Running, Role: LEADER
I20260929 20:58:09.948004 21459 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:09.947908 21464 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7134106c43844d0a617e47d26d12b9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40389 } }
I20260929 20:58:09.948513 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:58:09.951774 21465 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7134106c43844d0a617e47d26d12b9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7134106c43844d0a617e47d26d12b9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40389 } } }
I20260929 20:58:09.952370 21465 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:09.952889 21466 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7134106c43844d0a617e47d26d12b9f. Latest consensus state: current_term: 1 leader_uuid: "a7134106c43844d0a617e47d26d12b9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7134106c43844d0a617e47d26d12b9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40389 } } }
I20260929 20:58:09.953609 21466 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:09.955890 21475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:58:09.961822 21475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:58:09.975418 21475 catalog_manager.cc:1347] Generated new cluster ID: 5463486dceb440b38c3b1f9cd3ae5524
I20260929 20:58:09.976150 21475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:58:09.982690 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:09.956577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:35487 (local address 127.0.0.1:40389)
0929 20:58:09.956776 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:09.956782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:09.963945 (+ 7163us) server_negotiation.cc:407] Connection header received
0929 20:58:09.964443 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:09.964449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:09.964530 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:09.964610 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:09.966695 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:09.967840 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:09.969561 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:09.970334 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:09.970436 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:09.973953 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:09.973961 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:09.973967 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:09.974013 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:09.978500 (+ 4487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:09.979796 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:09.979802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:09.979806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:09.979974 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:09.980541 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:09.980547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:09.980550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:09.981023 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:09.981291 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:09.982028 (+ 737us) server_negotiation.cc:299] Negotiation successful
0929 20:58:09.982275 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:58:10.006063 21475 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:58:10.007664 21475 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:58:10.017288 21475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7134106c43844d0a617e47d26d12b9f: Generated new TSK 0
I20260929 20:58:10.018142 21475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:58:10.029532 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 21392
I20260929 20:58:10.031003 21454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:58:10.139407 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 21392
W20260929 20:58:10.153766 21377 connection.cc:535] client connection to 127.0.0.1:40389 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40389): Connection reset by peer (error 104)
2026-09-29T20:58:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1925 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:58:10 dist-test-slave-m9g1 krb5kdc[21491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:58:10 dist-test-slave-m9g1 krb5kdc[21491](info): set up 2 sockets
Sep 29 20:58:10 dist-test-slave-m9g1 krb5kdc[21491](info): commencing operation
krb5kdc: starting...
W20260929 20:58:11.170213 388 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:58:11 dist-test-slave-m9g1 krb5kdc[21491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:58:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:58:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:58:11.411247 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39043
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:58:11.730290 21504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:11.730603 21504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:11.730749 21504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:11.737856 21504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:11.737980 21504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:11.738060 21504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:11.738137 21504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:11.738210 21504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:11.747996 21504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:11.749282 21504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:11.750608 21504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:11.762145 21509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:11.762161 21512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:11.762207 21510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:11.762966 21504 server_base.cc:1034] running on GCE node
I20260929 20:58:11.763695 21504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:11.765100 21504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:11.766300 21504 hybrid_clock.cc:648] HybridClock initialized: now 1790715491766253 us; error 39 us; skew 500 ppm
I20260929 20:58:11.766636 21504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:11 dist-test-slave-m9g1 krb5kdc[21491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:11.769748 21504 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:58:11.771220 21504 webserver.cc:460] Webserver started at http://127.0.0.1:36181/ using document root <none> and password file <none>
I20260929 20:58:11.771915 21504 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:11.772018 21504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:11.772306 21504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:11.775259 21504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "eb7af0c72a3449d3a2442a0460e11a5f"
format_stamp: "Formatted at 2026-09-29 20:58:11 on dist-test-slave-m9g1"
server_key: "51947b50dfbf16c65252fe250163949d"
server_key_iv: "704f3007f444e4696ec6fef44ced7e20"
server_key_version: "encryptionkey@0"
I20260929 20:58:11.775990 21504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "eb7af0c72a3449d3a2442a0460e11a5f"
format_stamp: "Formatted at 2026-09-29 20:58:11 on dist-test-slave-m9g1"
server_key: "51947b50dfbf16c65252fe250163949d"
server_key_iv: "704f3007f444e4696ec6fef44ced7e20"
server_key_version: "encryptionkey@0"
I20260929 20:58:11.781069 21504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:58:11.784477 21519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:11.785795 21504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:58:11.785953 21504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "eb7af0c72a3449d3a2442a0460e11a5f"
format_stamp: "Formatted at 2026-09-29 20:58:11 on dist-test-slave-m9g1"
server_key: "51947b50dfbf16c65252fe250163949d"
server_key_iv: "704f3007f444e4696ec6fef44ced7e20"
server_key_version: "encryptionkey@0"
I20260929 20:58:11.786129 21504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:11.802026 21504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:11.806115 21504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:11.806391 21504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:11.822996 21504 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39043
I20260929 20:58:11.823027 21570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39043 every 8 connection(s)
I20260929 20:58:11.824280 21504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:58:11.827999 21571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:11.833954 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 21504
I20260929 20:58:11.834138 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:58:11.834692 388 external_mini_cluster.cc:1428] Setting key 7bbe517af5953cec7878d40f2b49beb7
I20260929 20:58:11.838632 21571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f: Bootstrap starting.
I20260929 20:58:11.841873 21571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:11.843103 21571 log.cc:824] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:11.846280 21571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f: No bootstrap required, opened a new log
I20260929 20:58:11.847973 21574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:11.836430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:51423 (local address 127.0.0.1:39043)
0929 20:58:11.837194 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:11.837205 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:11.837245 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:58:11.838956 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:11.838977 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:11.839367 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:11.839673 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:11.840261 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:11.841543 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:11.843421 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:11.844450 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:11.844581 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:11.845748 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:11.845755 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:11.845768 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:11.846334 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:11.846418 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:11.846445 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:11.846551 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:11.846833 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 20:58:11.847006 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":347,"threads_started":1}
I20260929 20:58:11.851630 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:58:11.855008 21571 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7af0c72a3449d3a2442a0460e11a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39043 } }
I20260929 20:58:11.855504 21571 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:58:11.855618 21571 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7af0c72a3449d3a2442a0460e11a5f, State: Initialized, Role: FOLLOWER
I20260929 20:58:11.856252 21571 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7af0c72a3449d3a2442a0460e11a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39043 } }
I20260929 20:58:11.856516 21571 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:58:11.856669 21571 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:58:11.856895 21571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:11.859824 21571 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7af0c72a3449d3a2442a0460e11a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39043 } }
I20260929 20:58:11.860320 21571 leader_election.cc:302] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb7af0c72a3449d3a2442a0460e11a5f; no voters:
I20260929 20:58:11.860917 21571 leader_election.cc:288] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:58:11.861016 21581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:58:11.862047 21581 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f [term 1 LEADER]: Becoming Leader. State: Replica: eb7af0c72a3449d3a2442a0460e11a5f, State: Running, Role: LEADER
I20260929 20:58:11.862955 21581 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7af0c72a3449d3a2442a0460e11a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39043 } }
I20260929 20:58:11.863130 21571 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:11.863741 21574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:11.856686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:53653 (local address 127.0.0.1:39043)
0929 20:58:11.856869 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:11.856876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:11.857296 (+ 420us) server_negotiation.cc:407] Connection header received
0929 20:58:11.857449 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:11.857454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:11.857535 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:11.857639 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:11.858068 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:11.859173 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:11.860574 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:11.861277 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:11.861411 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:11.862895 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:11.862901 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:11.862906 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:11.862929 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:11.863001 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:11.863022 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:11.863165 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:11.863377 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 20:58:11.863432 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:58:11.865774 21582 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb7af0c72a3449d3a2442a0460e11a5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7af0c72a3449d3a2442a0460e11a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39043 } } }
I20260929 20:58:11.866122 21582 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:11.866600 21583 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb7af0c72a3449d3a2442a0460e11a5f. Latest consensus state: current_term: 1 leader_uuid: "eb7af0c72a3449d3a2442a0460e11a5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7af0c72a3449d3a2442a0460e11a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39043 } } }
I20260929 20:58:11.866909 21583 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:11.867076 21587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:58:11.878716 21587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:58:11.890048 21587 catalog_manager.cc:1347] Generated new cluster ID: 5574375640b04bcbbbba884591ce5e70
I20260929 20:58:11.890467 21587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:58:11.907498 21587 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:58:11.909374 21587 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:58:11.916957 21587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb7af0c72a3449d3a2442a0460e11a5f: Generated new TSK 0
I20260929 20:58:11.917958 21587 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:58:11.932129 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 21504
W20260929 20:58:11.933176 21566 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.113:51423
W20260929 20:58:11.933941 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 21504: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:58:11.934089 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 21504
W20260929 20:58:11.947659 21489 connection.cc:535] client connection to 127.0.0.1:39043 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39043): Connection reset by peer (error 104)
2026-09-29T20:58:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1793 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:58:12 dist-test-slave-m9g1 krb5kdc[21602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:58:12 dist-test-slave-m9g1 krb5kdc[21602](info): set up 2 sockets
Sep 29 20:58:12 dist-test-slave-m9g1 krb5kdc[21602](info): commencing operation
krb5kdc: starting...
W20260929 20:58:12.962381 388 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:58:13 dist-test-slave-m9g1 krb5kdc[21602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:58:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:58:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:58:13.206724 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38313
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:58:13.527801 21615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:13.528081 21615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:13.528224 21615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:13.535567 21615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:13.535681 21615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:13.535727 21615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:13.535808 21615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:13.535857 21615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:13.545730 21615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:13.547060 21615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:13.548338 21615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:13.559659 21620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:13.559605 21623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:13.559576 21621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:13.560477 21615 server_base.cc:1034] running on GCE node
I20260929 20:58:13.561305 21615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:13.562865 21615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:13.564073 21615 hybrid_clock.cc:648] HybridClock initialized: now 1790715493564030 us; error 33 us; skew 500 ppm
I20260929 20:58:13.564443 21615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:13 dist-test-slave-m9g1 krb5kdc[21602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:13.571116 21615 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:58:13.572716 21615 webserver.cc:460] Webserver started at http://127.0.0.1:34459/ using document root <none> and password file <none>
I20260929 20:58:13.573462 21615 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:13.573587 21615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:13.573899 21615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:13.577060 21615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "06cc284ee1a348f98fb38082301d89d2"
format_stamp: "Formatted at 2026-09-29 20:58:13 on dist-test-slave-m9g1"
server_key: "9ffec3a6e36d90727367a94e0024d511"
server_key_iv: "13fa4673e2a0866ae3e802c9d527b715"
server_key_version: "encryptionkey@0"
I20260929 20:58:13.577832 21615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "06cc284ee1a348f98fb38082301d89d2"
format_stamp: "Formatted at 2026-09-29 20:58:13 on dist-test-slave-m9g1"
server_key: "9ffec3a6e36d90727367a94e0024d511"
server_key_iv: "13fa4673e2a0866ae3e802c9d527b715"
server_key_version: "encryptionkey@0"
I20260929 20:58:13.583469 21615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 20:58:13.587157 21630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:13.588616 21615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:58:13.588791 21615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "06cc284ee1a348f98fb38082301d89d2"
format_stamp: "Formatted at 2026-09-29 20:58:13 on dist-test-slave-m9g1"
server_key: "9ffec3a6e36d90727367a94e0024d511"
server_key_iv: "13fa4673e2a0866ae3e802c9d527b715"
server_key_version: "encryptionkey@0"
I20260929 20:58:13.588981 21615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:13.594874 21615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:13.598862 21615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:13.599125 21615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:13.616878 21615 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38313
I20260929 20:58:13.616909 21681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38313 every 8 connection(s)
I20260929 20:58:13.618242 21615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:58:13.619302 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 21615
I20260929 20:58:13.619470 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:58:13.619982 388 external_mini_cluster.cc:1428] Setting key b5d4e98cc947ba58594d83642a0eff3b
I20260929 20:58:13.622663 21682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:13.627518 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:13.621760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:58097 (local address 127.0.0.1:38313)
0929 20:58:13.622527 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:13.622539 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:13.622578 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:58:13.624216 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:13.624237 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:13.624574 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:13.624916 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:13.625926 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:13.625932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:13.625945 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:13.625959 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:13.626026 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:13.626055 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:13.626258 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:13.626363 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 20:58:13.626555 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":337,"threads_started":1}
I20260929 20:58:13.631026 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:58:13.634771 21682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2: Bootstrap starting.
I20260929 20:58:13.638705 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:13.636197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:40191 (local address 127.0.0.1:38313)
0929 20:58:13.636350 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:13.636355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:13.636720 (+ 365us) server_negotiation.cc:407] Connection header received
0929 20:58:13.636856 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:13.636860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:13.636907 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:13.636982 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:13.637979 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:13.637983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:13.637986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:13.638003 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:13.638044 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:13.638059 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:13.638156 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:13.638369 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 20:58:13.638421 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 20:58:13.638913 21682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:13.640126 21682 log.cc:824] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:13.642797 21682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2: No bootstrap required, opened a new log
I20260929 20:58:13.649976 21682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cc284ee1a348f98fb38082301d89d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38313 } }
I20260929 20:58:13.650378 21682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:58:13.650480 21682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06cc284ee1a348f98fb38082301d89d2, State: Initialized, Role: FOLLOWER
I20260929 20:58:13.651057 21682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cc284ee1a348f98fb38082301d89d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38313 } }
I20260929 20:58:13.651259 21682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:58:13.651384 21682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:58:13.651557 21682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:13.653656 21682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cc284ee1a348f98fb38082301d89d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38313 } }
I20260929 20:58:13.654232 21682 leader_election.cc:302] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06cc284ee1a348f98fb38082301d89d2; no voters:
I20260929 20:58:13.654793 21682 leader_election.cc:288] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:58:13.654902 21692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:58:13.655512 21692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2 [term 1 LEADER]: Becoming Leader. State: Replica: 06cc284ee1a348f98fb38082301d89d2, State: Running, Role: LEADER
I20260929 20:58:13.655987 21692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cc284ee1a348f98fb38082301d89d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38313 } }
I20260929 20:58:13.656674 21682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:13.658520 21694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06cc284ee1a348f98fb38082301d89d2. Latest consensus state: current_term: 1 leader_uuid: "06cc284ee1a348f98fb38082301d89d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cc284ee1a348f98fb38082301d89d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38313 } } }
I20260929 20:58:13.658768 21693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06cc284ee1a348f98fb38082301d89d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cc284ee1a348f98fb38082301d89d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38313 } } }
I20260929 20:58:13.659071 21693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2 [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:13.659071 21694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2 [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:13.660845 21699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:58:13.668160 21699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:58:13.676913 21699 catalog_manager.cc:1347] Generated new cluster ID: 07714b2d1b2c450c9f7537c506e583b6
I20260929 20:58:13.677021 21699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:58:13.699360 21699 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:58:13.699904 21699 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:58:13.707821 21699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06cc284ee1a348f98fb38082301d89d2: Generated new TSK 0
I20260929 20:58:13.708966 21699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:58:13.715607 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 21615
W20260929 20:58:13.716732 21677 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.113:58097
W20260929 20:58:13.717461 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 21615: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:58:13.717602 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 21615
2026-09-29T20:58:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1783 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:58:13 dist-test-slave-m9g1 krb5kdc[21713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:58:13 dist-test-slave-m9g1 krb5kdc[21713](info): set up 2 sockets
Sep 29 20:58:13 dist-test-slave-m9g1 krb5kdc[21713](info): commencing operation
krb5kdc: starting...
W20260929 20:58:14.730412 388 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:58:14 dist-test-slave-m9g1 krb5kdc[21713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:58:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:58:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:58:14.972776 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:58:15.291653 21726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:15.291934 21726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:15.292071 21726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:15.299147 21726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:15.299278 21726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:15.299356 21726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:15.299436 21726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:15.299495 21726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:15.309197 21726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38547
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:15.310587 21726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:15.311911 21726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:15.323619 21731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:15.323619 21732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:15.323827 21734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:15.324160 21726 server_base.cc:1034] running on GCE node
I20260929 20:58:15.325150 21726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:15.326694 21726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:15.327895 21726 hybrid_clock.cc:648] HybridClock initialized: now 1790715495327847 us; error 41 us; skew 500 ppm
I20260929 20:58:15.328248 21726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:15 dist-test-slave-m9g1 krb5kdc[21713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:15.331440 21726 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:58:15.333021 21726 webserver.cc:460] Webserver started at http://127.0.0.1:44797/ using document root <none> and password file <none>
I20260929 20:58:15.333719 21726 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:15.333842 21726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:15.334136 21726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:15.337239 21726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "f356de0f723940fdaac6805735859e0c"
format_stamp: "Formatted at 2026-09-29 20:58:15 on dist-test-slave-m9g1"
server_key: "44f620a78fe7412558503e8e8cd54ab0"
server_key_iv: "d704fe8922627025831944789bfa0f53"
server_key_version: "encryptionkey@0"
I20260929 20:58:15.338020 21726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "f356de0f723940fdaac6805735859e0c"
format_stamp: "Formatted at 2026-09-29 20:58:15 on dist-test-slave-m9g1"
server_key: "44f620a78fe7412558503e8e8cd54ab0"
server_key_iv: "d704fe8922627025831944789bfa0f53"
server_key_version: "encryptionkey@0"
I20260929 20:58:15.343461 21726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:58:15.347025 21741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:15.348443 21726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:58:15.348589 21726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "f356de0f723940fdaac6805735859e0c"
format_stamp: "Formatted at 2026-09-29 20:58:15 on dist-test-slave-m9g1"
server_key: "44f620a78fe7412558503e8e8cd54ab0"
server_key_iv: "d704fe8922627025831944789bfa0f53"
server_key_version: "encryptionkey@0"
I20260929 20:58:15.348764 21726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:15.367798 21726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:15.372078 21726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:15.372320 21726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:15.389323 21726 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38547
I20260929 20:58:15.389355 21792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38547 every 8 connection(s)
I20260929 20:58:15.390729 21726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:58:15.394539 21793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:15.395759 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 21726
I20260929 20:58:15.395936 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:58:15.396454 388 external_mini_cluster.cc:1428] Setting key 6edc0a8da5cd6b0f727a14a4a6ff609a
I20260929 20:58:15.406746 21793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c: Bootstrap starting.
Sep 29 20:58:15 dist-test-slave-m9g1 krb5kdc[21713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:58:15.409876 21793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:15.411239 21793 log.cc:824] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:15.414827 21793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c: No bootstrap required, opened a new log
I20260929 20:58:15.419335 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:15.398316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:59995 (local address 127.0.0.1:38547)
0929 20:58:15.399040 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:15.399051 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:15.399089 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:58:15.400707 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:15.400732 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:15.401128 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:15.401477 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:15.402913 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:15.404182 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:15.405422 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:15.406061 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:15.406258 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:15.410313 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:15.410329 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:15.410350 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:15.410408 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:15.414310 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:15.415234 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:15.415243 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:15.415252 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:15.415476 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:15.416019 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:15.416024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:15.416027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:15.416747 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:15.417085 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:15.417521 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 20:58:15.417906 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":301,"threads_started":1}
I20260929 20:58:15.423583 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:58:15.424453 21793 raft_consensus.cc:348] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f356de0f723940fdaac6805735859e0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } }
I20260929 20:58:15.424795 21793 raft_consensus.cc:374] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:58:15.424886 21793 raft_consensus.cc:729] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f356de0f723940fdaac6805735859e0c, State: Initialized, Role: FOLLOWER
I20260929 20:58:15.425443 21793 consensus_queue.cc:260] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f356de0f723940fdaac6805735859e0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } }
I20260929 20:58:15.425655 21793 raft_consensus.cc:388] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:58:15.425776 21793 raft_consensus.cc:482] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:58:15.425911 21793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:15.428299 21793 raft_consensus.cc:504] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f356de0f723940fdaac6805735859e0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } }
I20260929 20:58:15.428777 21793 leader_election.cc:302] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f356de0f723940fdaac6805735859e0c; no voters:
I20260929 20:58:15.429378 21793 leader_election.cc:288] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:58:15.429461 21803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:58:15.430660 21803 raft_consensus.cc:686] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c [term 1 LEADER]: Becoming Leader. State: Replica: f356de0f723940fdaac6805735859e0c, State: Running, Role: LEADER
I20260929 20:58:15.431298 21803 consensus_queue.cc:237] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f356de0f723940fdaac6805735859e0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } }
I20260929 20:58:15.431674 21793 sys_catalog.cc:564] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:15.434533 21804 sys_catalog.cc:455] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f356de0f723940fdaac6805735859e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f356de0f723940fdaac6805735859e0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } } }
I20260929 20:58:15.435150 21804 sys_catalog.cc:458] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:15.440349 21805 sys_catalog.cc:455] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f356de0f723940fdaac6805735859e0c. Latest consensus state: current_term: 1 leader_uuid: "f356de0f723940fdaac6805735859e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f356de0f723940fdaac6805735859e0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } } }
I20260929 20:58:15.440723 21805 sys_catalog.cc:458] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:15.441375 21811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:58:15.444742 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:15.427941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:58945 (local address 127.0.0.1:38547)
0929 20:58:15.428115 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:15.428121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:15.428479 (+ 358us) server_negotiation.cc:407] Connection header received
0929 20:58:15.428652 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:15.428656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:15.428719 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:15.428807 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:15.430417 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:15.431419 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:15.432712 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:15.433385 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:15.433516 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:15.436826 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:15.436834 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:15.436839 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:15.436882 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:15.440586 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:15.441583 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:15.441588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:15.441591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:15.441721 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:15.442302 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:15.442306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:15.442309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:15.442699 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:15.442945 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:15.444210 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0929 20:58:15.444390 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:58:15.447211 21811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:58:15.457147 21811 catalog_manager.cc:1347] Generated new cluster ID: 16ba06856b7c45329e6bf24264090849
I20260929 20:58:15.457332 21811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:58:15.478389 21811 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:58:15.480221 21811 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:58:15.494359 21811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f356de0f723940fdaac6805735859e0c: Generated new TSK 0
I20260929 20:58:15.495335 21811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:58:15.510109 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 21726
I20260929 20:58:15.511430 21788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:58:15.604784 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 21726
W20260929 20:58:15.619110 21711 connection.cc:535] client connection to 127.0.0.1:38547 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38547): Connection reset by peer (error 104)
2026-09-29T20:58:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1887 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:58:15 dist-test-slave-m9g1 krb5kdc[21825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:58:15 dist-test-slave-m9g1 krb5kdc[21825](info): set up 2 sockets
Sep 29 20:58:15 dist-test-slave-m9g1 krb5kdc[21825](info): commencing operation
krb5kdc: starting...
W20260929 20:58:16.626411 388 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:58:16 dist-test-slave-m9g1 krb5kdc[21825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:58:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:58:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:58:16.871309 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:58:17.192759 21838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:17.193035 21838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:17.193153 21838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:17.200301 21838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:17.200412 21838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:17.200469 21838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:17.200552 21838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:17.200613 21838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:17.210345 21838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39371
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:17.211658 21838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:17.212962 21838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:17.224507 21846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:17.224606 21844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:17.224663 21843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:17.225140 21838 server_base.cc:1034] running on GCE node
I20260929 20:58:17.226073 21838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:17.227598 21838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:17.228802 21838 hybrid_clock.cc:648] HybridClock initialized: now 1790715497228748 us; error 49 us; skew 500 ppm
I20260929 20:58:17.229146 21838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:17 dist-test-slave-m9g1 krb5kdc[21825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:17.232304 21838 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:58:17.233764 21838 webserver.cc:460] Webserver started at http://127.0.0.1:45711/ using document root <none> and password file <none>
I20260929 20:58:17.234479 21838 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:17.234584 21838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:17.234840 21838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:17.237792 21838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "b64636e9fb0346b2b7fed488a349169d"
format_stamp: "Formatted at 2026-09-29 20:58:17 on dist-test-slave-m9g1"
server_key: "6c4cbc4798327da9cf70904ecb0d95e4"
server_key_iv: "7c09ac9128a78ba393c61dd78e60bd3e"
server_key_version: "encryptionkey@0"
I20260929 20:58:17.238601 21838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "b64636e9fb0346b2b7fed488a349169d"
format_stamp: "Formatted at 2026-09-29 20:58:17 on dist-test-slave-m9g1"
server_key: "6c4cbc4798327da9cf70904ecb0d95e4"
server_key_iv: "7c09ac9128a78ba393c61dd78e60bd3e"
server_key_version: "encryptionkey@0"
I20260929 20:58:17.243944 21838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:58:17.247519 21853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:17.248976 21838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 20:58:17.249141 21838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "b64636e9fb0346b2b7fed488a349169d"
format_stamp: "Formatted at 2026-09-29 20:58:17 on dist-test-slave-m9g1"
server_key: "6c4cbc4798327da9cf70904ecb0d95e4"
server_key_iv: "7c09ac9128a78ba393c61dd78e60bd3e"
server_key_version: "encryptionkey@0"
I20260929 20:58:17.249343 21838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:17.272398 21838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:17.276484 21838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:17.276723 21838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:17.293987 21838 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39371
I20260929 20:58:17.294005 21904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39371 every 8 connection(s)
I20260929 20:58:17.295342 21838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:58:17.298911 21905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:17.304860 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 21838
I20260929 20:58:17.305019 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:58:17.305403 388 external_mini_cluster.cc:1428] Setting key 4666966db2185783e55aba64e127bfce
I20260929 20:58:17.309690 21905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d: Bootstrap starting.
I20260929 20:58:17.313628 21905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:17.314733 21905 log.cc:824] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:17.317894 21905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d: No bootstrap required, opened a new log
I20260929 20:58:17.320371 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:17.307193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:45255 (local address 127.0.0.1:39371)
0929 20:58:17.307939 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:17.307954 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:17.308002 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:58:17.310267 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:17.310296 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:17.310808 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:17.311243 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:17.311798 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:17.313698 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:17.315268 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:17.316067 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:17.316265 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:17.317502 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:17.317513 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:17.317532 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:17.318270 (+ 738us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:17.318362 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:17.318406 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:17.318575 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:17.318827 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 20:58:17.319001 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":304,"threads_started":1}
I20260929 20:58:17.324452 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:58:17.327943 21905 raft_consensus.cc:348] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64636e9fb0346b2b7fed488a349169d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39371 } }
I20260929 20:58:17.328465 21905 raft_consensus.cc:374] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:58:17.328567 21905 raft_consensus.cc:729] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64636e9fb0346b2b7fed488a349169d, State: Initialized, Role: FOLLOWER
I20260929 20:58:17.329257 21905 consensus_queue.cc:260] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64636e9fb0346b2b7fed488a349169d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39371 } }
I20260929 20:58:17.329500 21905 raft_consensus.cc:388] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:58:17.329668 21905 raft_consensus.cc:482] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:58:17.329878 21905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:17.332762 21905 raft_consensus.cc:504] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64636e9fb0346b2b7fed488a349169d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39371 } }
I20260929 20:58:17.333369 21905 leader_election.cc:302] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b64636e9fb0346b2b7fed488a349169d; no voters:
I20260929 20:58:17.334065 21905 leader_election.cc:288] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:58:17.334178 21915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:58:17.336212 21905 sys_catalog.cc:564] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:17.337772 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:17.329743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:52061 (local address 127.0.0.1:39371)
0929 20:58:17.329925 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:17.329931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:17.330469 (+ 538us) server_negotiation.cc:407] Connection header received
0929 20:58:17.330665 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:17.330669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:17.330742 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:17.330825 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:17.331414 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:17.332205 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:17.334078 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:17.334926 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:17.335078 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:17.336881 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:17.336888 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:17.336892 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:17.336918 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:17.336969 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:17.336992 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:17.337120 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:17.337288 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 20:58:17.337346 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 20:58:17.341068 21915 raft_consensus.cc:686] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d [term 1 LEADER]: Becoming Leader. State: Replica: b64636e9fb0346b2b7fed488a349169d, State: Running, Role: LEADER
I20260929 20:58:17.341573 21915 consensus_queue.cc:237] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64636e9fb0346b2b7fed488a349169d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39371 } }
I20260929 20:58:17.344429 21917 sys_catalog.cc:455] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b64636e9fb0346b2b7fed488a349169d. Latest consensus state: current_term: 1 leader_uuid: "b64636e9fb0346b2b7fed488a349169d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64636e9fb0346b2b7fed488a349169d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39371 } } }
I20260929 20:58:17.344810 21917 sys_catalog.cc:458] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:17.345243 21916 sys_catalog.cc:455] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b64636e9fb0346b2b7fed488a349169d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64636e9fb0346b2b7fed488a349169d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39371 } } }
I20260929 20:58:17.345481 21916 sys_catalog.cc:458] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:17.346575 21926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:58:17.351023 21926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:58:17.359145 21926 catalog_manager.cc:1347] Generated new cluster ID: ade945f6694346e6a2a37f5b6624f425
I20260929 20:58:17.359277 21926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:58:17.373167 21926 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:58:17.374464 21926 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:58:17.384773 21926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b64636e9fb0346b2b7fed488a349169d: Generated new TSK 0
I20260929 20:58:17.385612 21926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:58:17.396790 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 21838
W20260929 20:58:17.397994 21900 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.113:45255
W20260929 20:58:17.398708 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 21838: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:58:17.398861 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 21838
W20260929 20:58:17.417788 21823 connection.cc:535] client connection to 127.0.0.1:39371 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39371): Connection reset by peer (error 104)
2026-09-29T20:58:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1798 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:58:17 dist-test-slave-m9g1 krb5kdc[21936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:58:17 dist-test-slave-m9g1 krb5kdc[21936](info): set up 2 sockets
Sep 29 20:58:17 dist-test-slave-m9g1 krb5kdc[21936](info): commencing operation
krb5kdc: starting...
W20260929 20:58:18.410388 388 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:58:18 dist-test-slave-m9g1 krb5kdc[21936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:58:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:58:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:58:18.649477 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36783
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:58:18.969089 21949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:18.969388 21949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:18.969507 21949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:18.976785 21949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:18.976904 21949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:18.976958 21949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:18.977041 21949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:18.977093 21949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:18.986876 21949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36783
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:18.988113 21949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:18.989380 21949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:19.000618 21955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:19.000707 21954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:19.000593 21957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:19.001210 21949 server_base.cc:1034] running on GCE node
I20260929 20:58:19.002020 21949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:19.003464 21949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:19.004679 21949 hybrid_clock.cc:648] HybridClock initialized: now 1790715499004639 us; error 50 us; skew 500 ppm
I20260929 20:58:19.005038 21949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:19 dist-test-slave-m9g1 krb5kdc[21936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:19.014382 21949 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:58:19.015959 21949 webserver.cc:460] Webserver started at http://127.0.0.1:41429/ using document root <none> and password file <none>
I20260929 20:58:19.016674 21949 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:19.016789 21949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:19.017113 21949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:19.020349 21949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "9a368601cc2f4415925db0c79c047068"
format_stamp: "Formatted at 2026-09-29 20:58:19 on dist-test-slave-m9g1"
server_key: "9da31c49be2b0dfb2046272f05b2384a"
server_key_iv: "89451bf79cf4ac24114f5ab0936f8042"
server_key_version: "encryptionkey@0"
I20260929 20:58:19.021215 21949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "9a368601cc2f4415925db0c79c047068"
format_stamp: "Formatted at 2026-09-29 20:58:19 on dist-test-slave-m9g1"
server_key: "9da31c49be2b0dfb2046272f05b2384a"
server_key_iv: "89451bf79cf4ac24114f5ab0936f8042"
server_key_version: "encryptionkey@0"
I20260929 20:58:19.026724 21949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 20:58:19.030403 21964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:19.031982 21949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:58:19.032155 21949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "9a368601cc2f4415925db0c79c047068"
format_stamp: "Formatted at 2026-09-29 20:58:19 on dist-test-slave-m9g1"
server_key: "9da31c49be2b0dfb2046272f05b2384a"
server_key_iv: "89451bf79cf4ac24114f5ab0936f8042"
server_key_version: "encryptionkey@0"
I20260929 20:58:19.032348 21949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:19.038277 21949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:19.042759 21949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:19.043030 21949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:19.060187 21949 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36783
I20260929 20:58:19.060222 22015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36783 every 8 connection(s)
I20260929 20:58:19.061725 21949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:58:19.062111 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 21949
I20260929 20:58:19.062312 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:58:19.062932 388 external_mini_cluster.cc:1428] Setting key b7893663940127d10a6c0d052f981260
I20260929 20:58:19.067255 22016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:19.077586 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:19.071828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:50165 (local address 127.0.0.1:36783)
0929 20:58:19.072600 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:19.072614 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:19.072657 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:58:19.074272 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:19.074301 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:19.074674 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:19.075023 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:19.076081 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:19.076088 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:19.076101 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:19.076125 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:19.076188 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:19.076208 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:19.076369 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:19.076466 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 20:58:19.076651 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":333,"threads_started":1}
I20260929 20:58:19.081468 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:58:19.081703 22016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068: Bootstrap starting.
I20260929 20:58:19.085572 22016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:19.086865 22016 log.cc:824] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:19.088840 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:19.086515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:43913 (local address 127.0.0.1:36783)
0929 20:58:19.086708 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:19.086713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:19.087015 (+ 302us) server_negotiation.cc:407] Connection header received
0929 20:58:19.087193 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:19.087196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:19.087243 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:19.087321 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:19.088275 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:19.088280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:19.088283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:19.088297 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:19.088333 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:19.088347 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:19.088429 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:19.088540 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 20:58:19.088585 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:58:19.090026 22016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068: No bootstrap required, opened a new log
I20260929 20:58:19.097282 22016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a368601cc2f4415925db0c79c047068" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36783 } }
I20260929 20:58:19.097666 22016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:58:19.097759 22016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a368601cc2f4415925db0c79c047068, State: Initialized, Role: FOLLOWER
I20260929 20:58:19.098356 22016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a368601cc2f4415925db0c79c047068" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36783 } }
I20260929 20:58:19.098601 22016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:58:19.098752 22016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:58:19.098932 22016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:19.101184 22016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a368601cc2f4415925db0c79c047068" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36783 } }
I20260929 20:58:19.101688 22016 leader_election.cc:302] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a368601cc2f4415925db0c79c047068; no voters:
I20260929 20:58:19.102305 22016 leader_election.cc:288] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:58:19.102416 22026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:58:19.103031 22026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068 [term 1 LEADER]: Becoming Leader. State: Replica: 9a368601cc2f4415925db0c79c047068, State: Running, Role: LEADER
I20260929 20:58:19.103618 22026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a368601cc2f4415925db0c79c047068" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36783 } }
I20260929 20:58:19.104279 22016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:19.106251 22027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a368601cc2f4415925db0c79c047068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a368601cc2f4415925db0c79c047068" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36783 } } }
I20260929 20:58:19.106626 22027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068 [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:19.108767 22028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a368601cc2f4415925db0c79c047068. Latest consensus state: current_term: 1 leader_uuid: "9a368601cc2f4415925db0c79c047068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a368601cc2f4415925db0c79c047068" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36783 } } }
I20260929 20:58:19.109076 22028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068 [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:19.110173 22033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:58:19.115566 22033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:58:19.125329 22033 catalog_manager.cc:1347] Generated new cluster ID: 6a9982aca09a41a3ad7d913f0e142059
I20260929 20:58:19.125448 22033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:58:19.135898 22033 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:58:19.136332 22033 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:58:19.151598 22033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a368601cc2f4415925db0c79c047068: Generated new TSK 0
I20260929 20:58:19.152488 22033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:58:19.159646 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 21949
W20260929 20:58:19.160698 22011 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.113:50165
W20260929 20:58:19.161373 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 21949: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:58:19.161500 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 21949
2026-09-29T20:58:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1756 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:58:19 dist-test-slave-m9g1 krb5kdc[22047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:58:19 dist-test-slave-m9g1 krb5kdc[22047](info): set up 2 sockets
Sep 29 20:58:19 dist-test-slave-m9g1 krb5kdc[22047](info): commencing operation
krb5kdc: starting...
W20260929 20:58:20.186175 388 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:58:20 dist-test-slave-m9g1 krb5kdc[22047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:58:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:58:20.423686 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43039
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:58:20.745092 22060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:20.745381 22060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:20.745508 22060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:20.752797 22060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:20.752904 22060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:20.752974 22060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:20.753052 22060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:20.753108 22060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:20.762915 22060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43039
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:20.764182 22060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:20.765475 22060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:20.776746 22068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:20.776952 22066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:20.776919 22065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:20.777837 22060 server_base.cc:1034] running on GCE node
I20260929 20:58:20.778688 22060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:20.780185 22060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:20.781397 22060 hybrid_clock.cc:648] HybridClock initialized: now 1790715500781343 us; error 41 us; skew 500 ppm
I20260929 20:58:20.781771 22060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:20 dist-test-slave-m9g1 krb5kdc[22047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:20.784840 22060 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:58:20.786355 22060 webserver.cc:460] Webserver started at http://127.0.97.62:35267/ using document root <none> and password file <none>
I20260929 20:58:20.787029 22060 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:20.787142 22060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:20.787417 22060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:20.790541 22060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "fe89a089a2a143c9966389c377838865"
format_stamp: "Formatted at 2026-09-29 20:58:20 on dist-test-slave-m9g1"
server_key: "521cb59d2321bba887d8d572be0e608c"
server_key_iv: "915f749ab28168e614c21c37c8d230b3"
server_key_version: "encryptionkey@0"
I20260929 20:58:20.791332 22060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "fe89a089a2a143c9966389c377838865"
format_stamp: "Formatted at 2026-09-29 20:58:20 on dist-test-slave-m9g1"
server_key: "521cb59d2321bba887d8d572be0e608c"
server_key_iv: "915f749ab28168e614c21c37c8d230b3"
server_key_version: "encryptionkey@0"
I20260929 20:58:20.796448 22060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:58:20.799794 22075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:20.801049 22060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:58:20.801208 22060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "fe89a089a2a143c9966389c377838865"
format_stamp: "Formatted at 2026-09-29 20:58:20 on dist-test-slave-m9g1"
server_key: "521cb59d2321bba887d8d572be0e608c"
server_key_iv: "915f749ab28168e614c21c37c8d230b3"
server_key_version: "encryptionkey@0"
I20260929 20:58:20.801380 22060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:20.820709 22060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:20.824798 22060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:20.825073 22060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:20.842432 22060 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43039
I20260929 20:58:20.842450 22126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43039 every 8 connection(s)
I20260929 20:58:20.843850 22060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:58:20.846805 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 22060
I20260929 20:58:20.846972 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:58:20.847508 388 external_mini_cluster.cc:1428] Setting key 78369fb7090b9182adf2ff5894244aa6
I20260929 20:58:20.848042 22127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:20.862560 22127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865: Bootstrap starting.
Sep 29 20:58:20 dist-test-slave-m9g1 krb5kdc[22047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:58:20.866381 22127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:20.867779 22127 log.cc:824] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:20.871318 22127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865: No bootstrap required, opened a new log
I20260929 20:58:20.873044 22130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:20.849356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37913 (local address 127.0.97.62:43039)
0929 20:58:20.850266 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:20.850284 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:20.850615 (+ 331us) server_negotiation.cc:407] Connection header received
0929 20:58:20.852850 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:20.852873 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:20.853313 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:20.853700 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:20.855441 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:20.856745 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:20.858543 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:20.859339 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:20.859498 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:20.864199 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:20.864211 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:20.864233 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:20.864283 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:20.868030 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:20.869012 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:20.869020 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:20.869027 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:20.869238 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:20.869749 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:20.869754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:20.869757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:20.870511 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:20.870822 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:20.871260 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 20:58:20.871637 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":371,"threads_started":1}
I20260929 20:58:20.877353 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:58:20.880249 22127 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe89a089a2a143c9966389c377838865" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43039 } }
I20260929 20:58:20.880616 22127 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:58:20.880713 22127 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe89a089a2a143c9966389c377838865, State: Initialized, Role: FOLLOWER
I20260929 20:58:20.881294 22127 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe89a089a2a143c9966389c377838865" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43039 } }
I20260929 20:58:20.881577 22127 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:58:20.881778 22127 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:58:20.881978 22127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:20.884363 22127 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe89a089a2a143c9966389c377838865" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43039 } }
I20260929 20:58:20.884862 22127 leader_election.cc:302] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe89a089a2a143c9966389c377838865; no voters:
I20260929 20:58:20.885469 22127 leader_election.cc:288] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:58:20.885573 22137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:58:20.887845 22127 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:20.892472 22137 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865 [term 1 LEADER]: Becoming Leader. State: Replica: fe89a089a2a143c9966389c377838865, State: Running, Role: LEADER
I20260929 20:58:20.893167 22137 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe89a089a2a143c9966389c377838865" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43039 } }
I20260929 20:58:20.895300 22130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:20.881646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44657 (local address 127.0.97.62:43039)
0929 20:58:20.881819 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:20.881826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:20.882251 (+ 425us) server_negotiation.cc:407] Connection header received
0929 20:58:20.882424 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:20.882429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:20.882506 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:20.882591 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:20.883913 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:20.884725 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:20.886172 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:20.886761 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:20.886836 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:20.889373 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:20.889378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:20.889381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:20.889413 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:20.892181 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:20.893020 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:20.893026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:20.893033 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:20.893156 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:20.893628 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:20.893633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:20.893636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:20.893994 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:20.894262 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:20.894762 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 20:58:20.894932 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:58:20.897281 22138 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe89a089a2a143c9966389c377838865" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe89a089a2a143c9966389c377838865" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43039 } } }
I20260929 20:58:20.897360 22139 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe89a089a2a143c9966389c377838865. Latest consensus state: current_term: 1 leader_uuid: "fe89a089a2a143c9966389c377838865" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe89a089a2a143c9966389c377838865" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43039 } } }
I20260929 20:58:20.897832 22138 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865 [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:20.897927 22139 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865 [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:20.904337 22144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:58:20.910084 22144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:58:20.920166 22144 catalog_manager.cc:1347] Generated new cluster ID: 9e51f375d39a47ea8a3cdd8fd3d82734
I20260929 20:58:20.920280 22144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:58:20.928890 22144 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:58:20.930305 22144 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:58:20.935992 22144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe89a089a2a143c9966389c377838865: Generated new TSK 0
I20260929 20:58:20.936779 22144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:58:20.945565 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 22060
I20260929 20:58:20.947121 22122 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:58:21.034809 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 22060
W20260929 20:58:21.048961 22045 connection.cc:535] client connection to 127.0.97.62:43039 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:43039): Connection reset by peer (error 104)
2026-09-29T20:58:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1874 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:58:21 dist-test-slave-m9g1 krb5kdc[22160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:58:21 dist-test-slave-m9g1 krb5kdc[22160](info): set up 2 sockets
Sep 29 20:58:21 dist-test-slave-m9g1 krb5kdc[22160](info): commencing operation
krb5kdc: starting...
W20260929 20:58:22.046453 388 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:58:22 dist-test-slave-m9g1 krb5kdc[22160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:58:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:58:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:58:22.283713 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35557
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:58:22.608052 22173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:22.608335 22173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:22.608459 22173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:22.615645 22173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:22.615754 22173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:22.615823 22173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:22.615896 22173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:22.615939 22173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:22.625622 22173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35557
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:22.626916 22173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:22.628232 22173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:22.639694 22181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:22.639671 22178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:22.639724 22179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:22.640602 22173 server_base.cc:1034] running on GCE node
I20260929 20:58:22.641341 22173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:22.642794 22173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:22.644004 22173 hybrid_clock.cc:648] HybridClock initialized: now 1790715502643952 us; error 40 us; skew 500 ppm
I20260929 20:58:22.644337 22173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:22 dist-test-slave-m9g1 krb5kdc[22160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:22.647399 22173 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:58:22.648963 22173 webserver.cc:460] Webserver started at http://127.0.97.62:41129/ using document root <none> and password file <none>
I20260929 20:58:22.649744 22173 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:22.649860 22173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:22.650146 22173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:22.653101 22173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "bba752928bf6480babd58b1313feb58f"
format_stamp: "Formatted at 2026-09-29 20:58:22 on dist-test-slave-m9g1"
server_key: "61801c42ed97d768730759564dd1cc2c"
server_key_iv: "8e24eb6fd3e4964422d9068a3cbb2e62"
server_key_version: "encryptionkey@0"
I20260929 20:58:22.653851 22173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "bba752928bf6480babd58b1313feb58f"
format_stamp: "Formatted at 2026-09-29 20:58:22 on dist-test-slave-m9g1"
server_key: "61801c42ed97d768730759564dd1cc2c"
server_key_iv: "8e24eb6fd3e4964422d9068a3cbb2e62"
server_key_version: "encryptionkey@0"
I20260929 20:58:22.658973 22173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 20:58:22.662570 22188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:22.663972 22173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:58:22.664148 22173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "bba752928bf6480babd58b1313feb58f"
format_stamp: "Formatted at 2026-09-29 20:58:22 on dist-test-slave-m9g1"
server_key: "61801c42ed97d768730759564dd1cc2c"
server_key_iv: "8e24eb6fd3e4964422d9068a3cbb2e62"
server_key_version: "encryptionkey@0"
I20260929 20:58:22.664330 22173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:22.688890 22173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:22.693161 22173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:22.693432 22173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:22.710814 22173 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35557
I20260929 20:58:22.710839 22239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35557 every 8 connection(s)
I20260929 20:58:22.712239 22173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:58:22.715878 22240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:22.721565 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 22173
I20260929 20:58:22.721741 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:58:22.722278 388 external_mini_cluster.cc:1428] Setting key 4baa3668c7bdfd42592d737c67fbe606
I20260929 20:58:22.726984 22240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f: Bootstrap starting.
I20260929 20:58:22.730453 22240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:22.731544 22240 log.cc:824] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:22.735116 22240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f: No bootstrap required, opened a new log
I20260929 20:58:22.735872 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:22.724211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37551 (local address 127.0.97.62:35557)
0929 20:58:22.725012 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:22.725025 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:22.725074 (+ 49us) server_negotiation.cc:407] Connection header received
0929 20:58:22.726800 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:22.726824 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:22.727268 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:22.727633 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:22.728171 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:22.729536 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:22.730950 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:22.731662 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:22.731822 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:22.733437 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:22.733445 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:22.733463 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:22.734084 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:22.734171 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:22.734239 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:22.734391 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:22.734627 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 20:58:22.734820 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":353,"threads_started":1}
I20260929 20:58:22.739549 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:58:22.744311 22240 raft_consensus.cc:348] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba752928bf6480babd58b1313feb58f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35557 } }
I20260929 20:58:22.744830 22240 raft_consensus.cc:374] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:58:22.744930 22240 raft_consensus.cc:729] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba752928bf6480babd58b1313feb58f, State: Initialized, Role: FOLLOWER
I20260929 20:58:22.745589 22240 consensus_queue.cc:260] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba752928bf6480babd58b1313feb58f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35557 } }
I20260929 20:58:22.745837 22240 raft_consensus.cc:388] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:58:22.745972 22240 raft_consensus.cc:482] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:58:22.746147 22240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:22.748370 22240 raft_consensus.cc:504] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba752928bf6480babd58b1313feb58f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35557 } }
I20260929 20:58:22.748852 22240 leader_election.cc:302] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bba752928bf6480babd58b1313feb58f; no voters:
I20260929 20:58:22.749686 22240 leader_election.cc:288] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:58:22.749809 22250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:58:22.750913 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:22.744624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45987 (local address 127.0.97.62:35557)
0929 20:58:22.744810 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:22.744820 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:22.745090 (+ 270us) server_negotiation.cc:407] Connection header received
0929 20:58:22.745319 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:22.745324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:22.745403 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:22.745507 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:22.746004 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:22.746833 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:22.748097 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:22.748547 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:22.748671 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:22.749877 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:22.749884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:22.749890 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:22.749912 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:22.749968 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:22.749990 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:22.750120 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:22.750468 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 20:58:22.750533 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:58:22.751929 22240 sys_catalog.cc:564] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:22.756476 22250 raft_consensus.cc:686] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f [term 1 LEADER]: Becoming Leader. State: Replica: bba752928bf6480babd58b1313feb58f, State: Running, Role: LEADER
I20260929 20:58:22.756976 22250 consensus_queue.cc:237] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba752928bf6480babd58b1313feb58f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35557 } }
I20260929 20:58:22.759851 22252 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bba752928bf6480babd58b1313feb58f. Latest consensus state: current_term: 1 leader_uuid: "bba752928bf6480babd58b1313feb58f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba752928bf6480babd58b1313feb58f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35557 } } }
I20260929 20:58:22.760203 22252 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:22.760622 22251 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bba752928bf6480babd58b1313feb58f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba752928bf6480babd58b1313feb58f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35557 } } }
I20260929 20:58:22.760891 22251 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f [sys.catalog]: This master's current role is: LEADER
W20260929 20:58:22.766240 22265 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:58:22.766358 22265 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:58:22.766470 22264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:58:22.768568 22264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:58:22.777231 22264 catalog_manager.cc:1347] Generated new cluster ID: 291f16148b634de49b75f44bc1b5a9fe
I20260929 20:58:22.777347 22264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:58:22.806107 22264 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:58:22.807971 22264 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:58:22.833158 22264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bba752928bf6480babd58b1313feb58f: Generated new TSK 0
I20260929 20:58:22.834379 22264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:58:22.848912 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 22173
W20260929 20:58:22.850098 22235 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37551
W20260929 20:58:22.850888 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 22173: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:58:22.851020 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 22173
W20260929 20:58:22.864849 22158 connection.cc:535] client connection to 127.0.97.62:35557 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:35557): Connection reset by peer (error 104)
2026-09-29T20:58:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1815 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:58:22 dist-test-slave-m9g1 krb5kdc[22271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:58:22 dist-test-slave-m9g1 krb5kdc[22271](info): set up 2 sockets
Sep 29 20:58:22 dist-test-slave-m9g1 krb5kdc[22271](info): commencing operation
krb5kdc: starting...
W20260929 20:58:23.866446 388 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:58:24 dist-test-slave-m9g1 krb5kdc[22271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:58:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:58:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:58:24.117825 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45431
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:58:24.439970 22284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:24.440271 22284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:24.440368 22284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:24.447557 22284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:24.447676 22284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:24.447731 22284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:24.447789 22284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:24.447831 22284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:24.458025 22284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45431
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:24.459364 22284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:24.460656 22284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:24.472715 22289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:24.472687 22292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:24.472740 22290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:24.473418 22284 server_base.cc:1034] running on GCE node
I20260929 20:58:24.474169 22284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:24.475598 22284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:24.476806 22284 hybrid_clock.cc:648] HybridClock initialized: now 1790715504476757 us; error 41 us; skew 500 ppm
I20260929 20:58:24.477154 22284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:24 dist-test-slave-m9g1 krb5kdc[22271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:24.484241 22284 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:58:24.485777 22284 webserver.cc:460] Webserver started at http://127.0.97.62:38783/ using document root <none> and password file <none>
I20260929 20:58:24.486515 22284 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:24.486615 22284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:24.486869 22284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:24.489863 22284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "001460be93c146e1b2ac25eec1ac4435"
format_stamp: "Formatted at 2026-09-29 20:58:24 on dist-test-slave-m9g1"
server_key: "3e13f34ebb6f12efbfde60ecd94143d0"
server_key_iv: "f2a26ddad23cba81cbb86be84bedbfd2"
server_key_version: "encryptionkey@0"
I20260929 20:58:24.490614 22284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "001460be93c146e1b2ac25eec1ac4435"
format_stamp: "Formatted at 2026-09-29 20:58:24 on dist-test-slave-m9g1"
server_key: "3e13f34ebb6f12efbfde60ecd94143d0"
server_key_iv: "f2a26ddad23cba81cbb86be84bedbfd2"
server_key_version: "encryptionkey@0"
I20260929 20:58:24.495609 22284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 20:58:24.499059 22299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:24.500373 22284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:58:24.500519 22284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "001460be93c146e1b2ac25eec1ac4435"
format_stamp: "Formatted at 2026-09-29 20:58:24 on dist-test-slave-m9g1"
server_key: "3e13f34ebb6f12efbfde60ecd94143d0"
server_key_iv: "f2a26ddad23cba81cbb86be84bedbfd2"
server_key_version: "encryptionkey@0"
I20260929 20:58:24.500694 22284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:24.506273 22284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:24.510290 22284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:24.510569 22284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:24.527088 22284 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45431
I20260929 20:58:24.527112 22350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45431 every 8 connection(s)
I20260929 20:58:24.528407 22284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:58:24.531471 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 22284
I20260929 20:58:24.531636 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:58:24.532203 388 external_mini_cluster.cc:1428] Setting key 1439d964914538c595f44ac6f36b69fa
I20260929 20:58:24.532433 22351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:24.540270 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:24.533921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38121 (local address 127.0.97.62:45431)
0929 20:58:24.534639 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:24.534650 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:24.534689 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:58:24.536869 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:24.536892 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:24.537274 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:24.537617 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:24.538732 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:24.538739 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:24.538752 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:24.538766 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:24.538834 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:24.538853 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:24.539017 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:24.539126 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 20:58:24.539320 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":320,"threads_started":1}
I20260929 20:58:24.543798 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:58:24.545814 22351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435: Bootstrap starting.
I20260929 20:58:24.549726 22351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:24.551051 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:24.548704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33327 (local address 127.0.97.62:45431)
0929 20:58:24.548837 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:24.548841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:24.549239 (+ 398us) server_negotiation.cc:407] Connection header received
0929 20:58:24.549400 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:24.549403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:24.549468 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:24.549544 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:24.550458 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:24.550462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:24.550465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:24.550479 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:24.550516 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:24.550530 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:24.550616 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:24.550742 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 20:58:24.550785 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 20:58:24.551033 22351 log.cc:824] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:24.553953 22351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435: No bootstrap required, opened a new log
I20260929 20:58:24.561285 22351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001460be93c146e1b2ac25eec1ac4435" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45431 } }
I20260929 20:58:24.561671 22351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:58:24.561770 22351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001460be93c146e1b2ac25eec1ac4435, State: Initialized, Role: FOLLOWER
I20260929 20:58:24.562387 22351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001460be93c146e1b2ac25eec1ac4435" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45431 } }
I20260929 20:58:24.562604 22351 raft_consensus.cc:388] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:58:24.562733 22351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:58:24.562876 22351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:24.565259 22351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001460be93c146e1b2ac25eec1ac4435" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45431 } }
I20260929 20:58:24.565757 22351 leader_election.cc:302] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 001460be93c146e1b2ac25eec1ac4435; no voters:
I20260929 20:58:24.566360 22351 leader_election.cc:288] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:58:24.566457 22361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:58:24.567068 22361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435 [term 1 LEADER]: Becoming Leader. State: Replica: 001460be93c146e1b2ac25eec1ac4435, State: Running, Role: LEADER
I20260929 20:58:24.567610 22361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001460be93c146e1b2ac25eec1ac4435" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45431 } }
I20260929 20:58:24.568323 22351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:24.570341 22362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "001460be93c146e1b2ac25eec1ac4435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001460be93c146e1b2ac25eec1ac4435" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45431 } } }
I20260929 20:58:24.570883 22363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 001460be93c146e1b2ac25eec1ac4435. Latest consensus state: current_term: 1 leader_uuid: "001460be93c146e1b2ac25eec1ac4435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001460be93c146e1b2ac25eec1ac4435" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45431 } } }
I20260929 20:58:24.571273 22363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435 [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:24.575730 22362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435 [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:24.578423 22370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:58:24.583226 22370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:58:24.592206 22370 catalog_manager.cc:1347] Generated new cluster ID: 1bc6edfd459a4ce4aa7c375bae94bfdd
I20260929 20:58:24.592310 22370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:58:24.614666 22370 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:58:24.615082 22370 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:58:24.621577 22370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 001460be93c146e1b2ac25eec1ac4435: Generated new TSK 0
I20260929 20:58:24.622426 22370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:58:24.632076 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 22284
W20260929 20:58:24.633118 22346 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38121
W20260929 20:58:24.633767 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 22284: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:58:24.633920 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 22284
2026-09-29T20:58:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1782 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:58:24 dist-test-slave-m9g1 krb5kdc[22382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:58:24 dist-test-slave-m9g1 krb5kdc[22382](info): set up 2 sockets
Sep 29 20:58:24 dist-test-slave-m9g1 krb5kdc[22382](info): commencing operation
krb5kdc: starting...
W20260929 20:58:25.638442 388 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:58:25 dist-test-slave-m9g1 krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:58:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:58:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:58:25.879909 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42235
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:58:26.203256 22395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:26.203570 22395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:26.203702 22395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:26.210978 22395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:26.211092 22395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:26.211165 22395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:26.211246 22395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:26.211302 22395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:26.221199 22395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:26.222493 22395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:26.223724 22395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:26.234999 22403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:26.235029 22401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:26.235258 22400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:26.235690 22395 server_base.cc:1034] running on GCE node
I20260929 20:58:26.236485 22395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:26.237818 22395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:26.239007 22395 hybrid_clock.cc:648] HybridClock initialized: now 1790715506238950 us; error 50 us; skew 500 ppm
I20260929 20:58:26.239339 22395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:26 dist-test-slave-m9g1 krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:26.242302 22395 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:58:26.243639 22395 webserver.cc:460] Webserver started at http://127.0.97.62:32773/ using document root <none> and password file <none>
I20260929 20:58:26.244295 22395 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:26.244402 22395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:26.244685 22395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:26.247761 22395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "b7496e2bf35f4dae9ab23542fb9d441f"
format_stamp: "Formatted at 2026-09-29 20:58:26 on dist-test-slave-m9g1"
server_key: "52c01230ac2d606b8a9f8875fb68d6b7"
server_key_iv: "ff8fadb51ae3075f60880af94e7f3601"
server_key_version: "encryptionkey@0"
I20260929 20:58:26.248590 22395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "b7496e2bf35f4dae9ab23542fb9d441f"
format_stamp: "Formatted at 2026-09-29 20:58:26 on dist-test-slave-m9g1"
server_key: "52c01230ac2d606b8a9f8875fb68d6b7"
server_key_iv: "ff8fadb51ae3075f60880af94e7f3601"
server_key_version: "encryptionkey@0"
I20260929 20:58:26.253772 22395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:58:26.257256 22410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:26.258628 22395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:58:26.258785 22395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "b7496e2bf35f4dae9ab23542fb9d441f"
format_stamp: "Formatted at 2026-09-29 20:58:26 on dist-test-slave-m9g1"
server_key: "52c01230ac2d606b8a9f8875fb68d6b7"
server_key_iv: "ff8fadb51ae3075f60880af94e7f3601"
server_key_version: "encryptionkey@0"
I20260929 20:58:26.258965 22395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:26.275023 22395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:26.278966 22395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:26.279219 22395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:26.296119 22395 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42235
I20260929 20:58:26.296149 22461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42235 every 8 connection(s)
I20260929 20:58:26.297422 22395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:58:26.301100 22462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:26.303258 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 22395
I20260929 20:58:26.303468 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:58:26.303865 388 external_mini_cluster.cc:1428] Setting key 78ea381a86074a41a0b5a25fd142fc9d
I20260929 20:58:26.312610 22462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f: Bootstrap starting.
I20260929 20:58:26.316380 22462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:26.317727 22462 log.cc:824] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:58:26 dist-test-slave-m9g1 krb5kdc[22382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:58:26.320827 22462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f: No bootstrap required, opened a new log
I20260929 20:58:26.326668 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:26.305575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39863 (local address 127.0.97.62:42235)
0929 20:58:26.306411 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:26.306423 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:26.306465 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:58:26.308152 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:26.308176 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:26.308588 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:26.308935 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:26.310668 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:26.312006 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:26.313751 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:26.314494 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:26.314631 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:26.319477 (+ 4846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:26.319485 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:26.319501 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:26.319538 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:26.322564 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:26.323219 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:26.323226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:26.323232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:26.323413 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:26.323779 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:26.323783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:26.323785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:26.324348 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:26.324638 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:26.325101 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 20:58:26.325533 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":366,"threads_started":1}
I20260929 20:58:26.328840 22462 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7496e2bf35f4dae9ab23542fb9d441f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42235 } }
I20260929 20:58:26.329257 22462 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:58:26.329365 22462 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7496e2bf35f4dae9ab23542fb9d441f, State: Initialized, Role: FOLLOWER
I20260929 20:58:26.329969 22462 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7496e2bf35f4dae9ab23542fb9d441f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42235 } }
I20260929 20:58:26.330276 22462 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:58:26.330432 22462 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:58:26.330600 22462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:26.330641 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:58:26.333647 22462 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7496e2bf35f4dae9ab23542fb9d441f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42235 } }
I20260929 20:58:26.334328 22462 leader_election.cc:302] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7496e2bf35f4dae9ab23542fb9d441f; no voters:
I20260929 20:58:26.334941 22462 leader_election.cc:288] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:58:26.335039 22471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:58:26.336539 22471 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f [term 1 LEADER]: Becoming Leader. State: Replica: b7496e2bf35f4dae9ab23542fb9d441f, State: Running, Role: LEADER
I20260929 20:58:26.337184 22462 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:26.337210 22471 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7496e2bf35f4dae9ab23542fb9d441f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42235 } }
I20260929 20:58:26.340840 22473 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7496e2bf35f4dae9ab23542fb9d441f. Latest consensus state: current_term: 1 leader_uuid: "b7496e2bf35f4dae9ab23542fb9d441f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7496e2bf35f4dae9ab23542fb9d441f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42235 } } }
I20260929 20:58:26.341231 22473 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:26.341697 22472 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7496e2bf35f4dae9ab23542fb9d441f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7496e2bf35f4dae9ab23542fb9d441f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42235 } } }
I20260929 20:58:26.341962 22472 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:26.353168 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:26.335763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52021 (local address 127.0.97.62:42235)
0929 20:58:26.335940 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:26.335946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:26.337170 (+ 1224us) server_negotiation.cc:407] Connection header received
0929 20:58:26.337408 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:26.337414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:26.337495 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:26.337597 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:26.339479 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:26.340631 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:26.342468 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:26.342922 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:26.343037 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:26.345810 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:26.345816 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:26.345821 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:26.345862 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:26.348888 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:26.349800 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:26.349807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:26.349811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:26.349986 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:26.350458 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:26.350464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:26.350467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:26.350881 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:26.351136 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:26.352355 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0929 20:58:26.352596 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:58:26.357352 22481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:58:26.362800 22481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:58:26.370867 22481 catalog_manager.cc:1347] Generated new cluster ID: e39592b734f2406b8f15beece8489886
I20260929 20:58:26.370986 22481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:58:26.392094 22481 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:58:26.393426 22481 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:58:26.400796 22481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7496e2bf35f4dae9ab23542fb9d441f: Generated new TSK 0
I20260929 20:58:26.401607 22481 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:58:26.411108 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 22395
I20260929 20:58:26.412184 22457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:58:26.502988 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 22395
2026-09-29T20:58:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:58:26 dist-test-slave-m9g1 krb5kdc[22494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:58:26 dist-test-slave-m9g1 krb5kdc[22494](info): set up 2 sockets
Sep 29 20:58:26 dist-test-slave-m9g1 krb5kdc[22494](info): commencing operation
krb5kdc: starting...
W20260929 20:58:27.506162 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:58:27 dist-test-slave-m9g1 krb5kdc[22494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:58:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:58:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:58:27.745755 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37603
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:58:28.068915 22507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:28.069211 22507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:28.069329 22507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:28.076725 22507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:28.076845 22507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:28.076920 22507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:28.076990 22507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:28.077035 22507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:28.087076 22507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:28.088387 22507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:28.089689 22507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:28.101037 22512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:28.101109 22515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:28.101150 22513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:28.101703 22507 server_base.cc:1034] running on GCE node
I20260929 20:58:28.102511 22507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:28.103870 22507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:28.105067 22507 hybrid_clock.cc:648] HybridClock initialized: now 1790715508105017 us; error 45 us; skew 500 ppm
I20260929 20:58:28.105392 22507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:28 dist-test-slave-m9g1 krb5kdc[22494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:28.108384 22507 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:58:28.109841 22507 webserver.cc:460] Webserver started at http://127.0.97.62:39691/ using document root <none> and password file <none>
I20260929 20:58:28.110570 22507 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:28.110690 22507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:28.110982 22507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:28.113873 22507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "aed7c89b904a45209ac8d40ffbc5de2e"
format_stamp: "Formatted at 2026-09-29 20:58:28 on dist-test-slave-m9g1"
server_key: "c199404daa6d3e6201afd890c542e571"
server_key_iv: "5a83d92fdcb1b2d4d21cfb1ec955480b"
server_key_version: "encryptionkey@0"
I20260929 20:58:28.114707 22507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "aed7c89b904a45209ac8d40ffbc5de2e"
format_stamp: "Formatted at 2026-09-29 20:58:28 on dist-test-slave-m9g1"
server_key: "c199404daa6d3e6201afd890c542e571"
server_key_iv: "5a83d92fdcb1b2d4d21cfb1ec955480b"
server_key_version: "encryptionkey@0"
I20260929 20:58:28.119642 22507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 20:58:28.122908 22522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:28.124151 22507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:58:28.124287 22507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "aed7c89b904a45209ac8d40ffbc5de2e"
format_stamp: "Formatted at 2026-09-29 20:58:28 on dist-test-slave-m9g1"
server_key: "c199404daa6d3e6201afd890c542e571"
server_key_iv: "5a83d92fdcb1b2d4d21cfb1ec955480b"
server_key_version: "encryptionkey@0"
I20260929 20:58:28.124452 22507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:28.142395 22507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:28.146643 22507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:28.146870 22507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:28.163921 22507 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37603
I20260929 20:58:28.163940 22573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37603 every 8 connection(s)
I20260929 20:58:28.165292 22507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:58:28.168341 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 22507
I20260929 20:58:28.168493 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:58:28.168890 388 external_mini_cluster.cc:1428] Setting key ebb36a67804714482b85f2baef68cf5b
I20260929 20:58:28.169296 22574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:28.182638 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:28.170720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50729 (local address 127.0.97.62:37603)
0929 20:58:28.171472 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:28.171484 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:28.171526 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:58:28.173856 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:28.173879 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:28.174326 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:28.174681 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:28.175272 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:28.176607 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:28.178344 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:28.178999 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:28.179139 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:28.180369 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:28.180378 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:28.180391 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:28.180958 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:28.181036 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:28.181061 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:28.181139 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:28.181354 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 20:58:28.181621 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":338,"threads_started":1}
I20260929 20:58:28.182897 22574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e: Bootstrap starting.
I20260929 20:58:28.186254 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:58:28.186381 22574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:28.187371 22574 log.cc:824] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:28.190373 22574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e: No bootstrap required, opened a new log
I20260929 20:58:28.196692 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:28.190494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55415 (local address 127.0.97.62:37603)
0929 20:58:28.190652 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:28.190658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:28.191036 (+ 378us) server_negotiation.cc:407] Connection header received
0929 20:58:28.191180 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:28.191185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:28.191258 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:28.191373 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:28.191862 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:28.192953 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:28.194157 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:28.194841 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:28.194996 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:28.195937 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:28.195944 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:28.195952 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:28.195973 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:28.196024 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:28.196044 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:28.196142 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:28.196235 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 20:58:28.196334 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 20:58:28.199644 22574 raft_consensus.cc:348] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed7c89b904a45209ac8d40ffbc5de2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37603 } }
I20260929 20:58:28.200035 22574 raft_consensus.cc:374] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:58:28.200119 22574 raft_consensus.cc:729] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aed7c89b904a45209ac8d40ffbc5de2e, State: Initialized, Role: FOLLOWER
I20260929 20:58:28.200667 22574 consensus_queue.cc:260] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed7c89b904a45209ac8d40ffbc5de2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37603 } }
I20260929 20:58:28.200928 22574 raft_consensus.cc:388] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:58:28.201076 22574 raft_consensus.cc:482] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:58:28.201246 22574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:28.203456 22574 raft_consensus.cc:504] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed7c89b904a45209ac8d40ffbc5de2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37603 } }
I20260929 20:58:28.203928 22574 leader_election.cc:302] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aed7c89b904a45209ac8d40ffbc5de2e; no voters:
I20260929 20:58:28.204483 22574 leader_election.cc:288] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:58:28.204610 22584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:58:28.205233 22584 raft_consensus.cc:686] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e [term 1 LEADER]: Becoming Leader. State: Replica: aed7c89b904a45209ac8d40ffbc5de2e, State: Running, Role: LEADER
I20260929 20:58:28.205788 22584 consensus_queue.cc:237] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed7c89b904a45209ac8d40ffbc5de2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37603 } }
I20260929 20:58:28.206471 22574 sys_catalog.cc:564] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:28.208348 22585 sys_catalog.cc:455] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aed7c89b904a45209ac8d40ffbc5de2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed7c89b904a45209ac8d40ffbc5de2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37603 } } }
I20260929 20:58:28.208725 22585 sys_catalog.cc:458] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:28.209756 22586 sys_catalog.cc:455] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader aed7c89b904a45209ac8d40ffbc5de2e. Latest consensus state: current_term: 1 leader_uuid: "aed7c89b904a45209ac8d40ffbc5de2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed7c89b904a45209ac8d40ffbc5de2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37603 } } }
I20260929 20:58:28.210067 22586 sys_catalog.cc:458] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:28.211310 22590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:58:28.216521 22590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:58:28.225458 22590 catalog_manager.cc:1347] Generated new cluster ID: ada3898b7b4f45d3b378a76a5afb2c07
I20260929 20:58:28.225576 22590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:58:28.235491 22590 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:58:28.236824 22590 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:58:28.253079 22590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aed7c89b904a45209ac8d40ffbc5de2e: Generated new TSK 0
I20260929 20:58:28.254279 22590 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:58:28.263489 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 22507
W20260929 20:58:28.264565 22569 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50729
W20260929 20:58:28.265304 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 22507: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:58:28.265455 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 22507
2026-09-29T20:58:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1761 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:58:28 dist-test-slave-m9g1 krb5kdc[22605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:58:28 dist-test-slave-m9g1 krb5kdc[22605](info): set up 2 sockets
Sep 29 20:58:28 dist-test-slave-m9g1 krb5kdc[22605](info): commencing operation
krb5kdc: starting...
W20260929 20:58:29.270403 388 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:58:29 dist-test-slave-m9g1 krb5kdc[22605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:58:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:58:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:58:29.513146 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44021
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:58:29.839421 22618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:29.839706 22618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:29.839823 22618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:29.846990 22618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:29.847100 22618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:29.847172 22618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:29.847254 22618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:29.847312 22618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:29.857097 22618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:29.858387 22618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:29.859722 22618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:29.871414 22623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:29.871507 22626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:29.871482 22624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:29.872131 22618 server_base.cc:1034] running on GCE node
I20260929 20:58:29.872923 22618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:29.874362 22618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:29.875593 22618 hybrid_clock.cc:648] HybridClock initialized: now 1790715509875525 us; error 54 us; skew 500 ppm
I20260929 20:58:29.875955 22618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:29 dist-test-slave-m9g1 krb5kdc[22605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:29.882642 22618 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:58:29.884060 22618 webserver.cc:460] Webserver started at http://127.0.97.62:36539/ using document root <none> and password file <none>
I20260929 20:58:29.884725 22618 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:29.884824 22618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:29.885075 22618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:29.888069 22618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "f70b7bace3204f8385220e8d095ebc1a"
format_stamp: "Formatted at 2026-09-29 20:58:29 on dist-test-slave-m9g1"
server_key: "1d600c77d78483923c51e9ff179cd8e1"
server_key_iv: "236d2a788056a64a80003b9ae420024a"
server_key_version: "encryptionkey@0"
I20260929 20:58:29.888772 22618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "f70b7bace3204f8385220e8d095ebc1a"
format_stamp: "Formatted at 2026-09-29 20:58:29 on dist-test-slave-m9g1"
server_key: "1d600c77d78483923c51e9ff179cd8e1"
server_key_iv: "236d2a788056a64a80003b9ae420024a"
server_key_version: "encryptionkey@0"
I20260929 20:58:29.893934 22618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:58:29.897521 22633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:29.898983 22618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:58:29.899142 22618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "f70b7bace3204f8385220e8d095ebc1a"
format_stamp: "Formatted at 2026-09-29 20:58:29 on dist-test-slave-m9g1"
server_key: "1d600c77d78483923c51e9ff179cd8e1"
server_key_iv: "236d2a788056a64a80003b9ae420024a"
server_key_version: "encryptionkey@0"
I20260929 20:58:29.899319 22618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:29.904871 22618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:29.908921 22618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:29.909184 22618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:29.925988 22618 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44021
I20260929 20:58:29.926023 22684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44021 every 8 connection(s)
I20260929 20:58:29.927412 22618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:58:29.931392 22685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:29.936710 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 22618
I20260929 20:58:29.936887 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:58:29.937423 388 external_mini_cluster.cc:1428] Setting key 374a265dfdaea9b8167bc3d53db6f2cb
I20260929 20:58:29.944644 22685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a: Bootstrap starting.
I20260929 20:58:29.946712 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:29.939237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45065 (local address 127.0.97.62:44021)
0929 20:58:29.940058 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:29.940078 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:29.940132 (+ 54us) server_negotiation.cc:407] Connection header received
0929 20:58:29.942787 (+ 2655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:29.942816 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:29.943285 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:29.943750 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:29.944725 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:29.944734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:29.944750 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:29.944768 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:29.944848 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:29.944874 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:29.945096 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:29.945234 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 20:58:29.945445 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":323,"threads_started":1}
I20260929 20:58:29.948334 22685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:29.949558 22685 log.cc:824] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:29.950614 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:58:29.952394 22685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a: No bootstrap required, opened a new log
I20260929 20:58:29.957104 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:29.954635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35069 (local address 127.0.97.62:44021)
0929 20:58:29.954832 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:29.954839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:29.955305 (+ 466us) server_negotiation.cc:407] Connection header received
0929 20:58:29.955451 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:29.955454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:29.955511 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:29.955605 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:29.956491 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:29.956495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:29.956498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:58:29.956512 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:29.956548 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:58:29.956561 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:29.956643 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:29.956761 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 20:58:29.956809 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:58:29.960960 22685 raft_consensus.cc:348] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70b7bace3204f8385220e8d095ebc1a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44021 } }
I20260929 20:58:29.961318 22685 raft_consensus.cc:374] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:58:29.961390 22685 raft_consensus.cc:729] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f70b7bace3204f8385220e8d095ebc1a, State: Initialized, Role: FOLLOWER
I20260929 20:58:29.961925 22685 consensus_queue.cc:260] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70b7bace3204f8385220e8d095ebc1a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44021 } }
I20260929 20:58:29.962141 22685 raft_consensus.cc:388] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:58:29.962268 22685 raft_consensus.cc:482] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:58:29.962406 22685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:29.964692 22685 raft_consensus.cc:504] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70b7bace3204f8385220e8d095ebc1a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44021 } }
I20260929 20:58:29.965185 22685 leader_election.cc:302] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f70b7bace3204f8385220e8d095ebc1a; no voters:
I20260929 20:58:29.965687 22685 leader_election.cc:288] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:58:29.965806 22695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:58:29.966485 22695 raft_consensus.cc:686] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a [term 1 LEADER]: Becoming Leader. State: Replica: f70b7bace3204f8385220e8d095ebc1a, State: Running, Role: LEADER
I20260929 20:58:29.966996 22695 consensus_queue.cc:237] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70b7bace3204f8385220e8d095ebc1a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44021 } }
I20260929 20:58:29.967638 22685 sys_catalog.cc:564] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:29.969904 22697 sys_catalog.cc:455] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f70b7bace3204f8385220e8d095ebc1a. Latest consensus state: current_term: 1 leader_uuid: "f70b7bace3204f8385220e8d095ebc1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70b7bace3204f8385220e8d095ebc1a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44021 } } }
I20260929 20:58:29.969935 22696 sys_catalog.cc:455] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f70b7bace3204f8385220e8d095ebc1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70b7bace3204f8385220e8d095ebc1a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44021 } } }
I20260929 20:58:29.970395 22697 sys_catalog.cc:458] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:29.970425 22696 sys_catalog.cc:458] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:29.971660 22700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:58:29.976300 22700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:58:29.984434 22700 catalog_manager.cc:1347] Generated new cluster ID: 83b5c85218e94171aa0b086c1ccc660b
I20260929 20:58:29.984582 22700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:58:29.998522 22700 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:58:29.998992 22700 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:58:30.015806 22700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f70b7bace3204f8385220e8d095ebc1a: Generated new TSK 0
I20260929 20:58:30.016960 22700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:58:30.024819 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 22618
W20260929 20:58:30.025889 22680 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45065
W20260929 20:58:30.026616 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 22618: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:58:30.026751 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 22618
2026-09-29T20:58:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1761 ms)
[----------] 18 tests from AuthTokenIssuingTest (32698 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:58:30 dist-test-slave-m9g1 krb5kdc[22716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:58:30 dist-test-slave-m9g1 krb5kdc[22716](info): set up 2 sockets
Sep 29 20:58:30 dist-test-slave-m9g1 krb5kdc[22716](info): commencing operation
krb5kdc: starting...
W20260929 20:58:31.058533 388 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:58:31 dist-test-slave-m9g1 krb5kdc[22716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:58:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:58:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:58:31.303287 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43927
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43927,127.0.97.61:34009,127.0.97.60:35711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:58:31.622946 22729 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:58:31.623323 22729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:31.623435 22729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:31.623562 22729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:31.631247 22729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:31.631377 22729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:31.631470 22729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:31.631537 22729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:31.631620 22729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:31.641584 22729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43927,127.0.97.61:34009,127.0.97.60:35711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:31.642964 22729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:31.644306 22729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:31.655742 22734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:31.655771 22735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:31.655920 22737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:31.656473 22729 server_base.cc:1034] running on GCE node
I20260929 20:58:31.657222 22729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:31.658666 22729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:31.659876 22729 hybrid_clock.cc:648] HybridClock initialized: now 1790715511659806 us; error 44 us; skew 500 ppm
I20260929 20:58:31.660214 22729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:31 dist-test-slave-m9g1 krb5kdc[22716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:31.663300 22729 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:58:31.664817 22729 webserver.cc:460] Webserver started at http://127.0.97.62:43685/ using document root <none> and password file <none>
I20260929 20:58:31.665501 22729 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:31.665601 22729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:31.665898 22729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:31.668967 22729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "15538a1d52cb44389c15aa2e5a8306eb"
format_stamp: "Formatted at 2026-09-29 20:58:31 on dist-test-slave-m9g1"
server_key: "ff62050d3ec0084ac06ec3c82af09339"
server_key_iv: "1883f90cf02a9f30223ebd5e2c0b5d5d"
server_key_version: "encryptionkey@0"
I20260929 20:58:31.669714 22729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "15538a1d52cb44389c15aa2e5a8306eb"
format_stamp: "Formatted at 2026-09-29 20:58:31 on dist-test-slave-m9g1"
server_key: "ff62050d3ec0084ac06ec3c82af09339"
server_key_iv: "1883f90cf02a9f30223ebd5e2c0b5d5d"
server_key_version: "encryptionkey@0"
I20260929 20:58:31.674805 22729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 20:58:31.678153 22744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:31.679533 22729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:58:31.679692 22729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "15538a1d52cb44389c15aa2e5a8306eb"
format_stamp: "Formatted at 2026-09-29 20:58:31 on dist-test-slave-m9g1"
server_key: "ff62050d3ec0084ac06ec3c82af09339"
server_key_iv: "1883f90cf02a9f30223ebd5e2c0b5d5d"
server_key_version: "encryptionkey@0"
I20260929 20:58:31.679886 22729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:31.699927 22729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:31.703795 22729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:31.704041 22729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:31.721314 22729 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43927
I20260929 20:58:31.721321 22795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43927 every 8 connection(s)
I20260929 20:58:31.722674 22729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:58:31.725170 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 22729
I20260929 20:58:31.725376 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:58:31.726150 388 external_mini_cluster.cc:1428] Setting key d5482f2714ea2260ea44e9e200dab913
I20260929 20:58:31.726925 22796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:31.735431 22796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:58:31 dist-test-slave-m9g1 krb5kdc[22716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260929 20:58:31.757687 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:31.740101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39128 (local address 127.0.97.62:43927)
0929 20:58:31.740743 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:31.740751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:31.740797 (+ 46us) server_negotiation.cc:407] Connection header received
0929 20:58:31.742042 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:31.742051 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:31.742365 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:31.742577 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:31.743716 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:31.745348 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:31.746732 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:31.747551 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:31.747716 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:31.751164 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:31.751174 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:31.751191 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:31.751228 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:31.753841 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:31.754589 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:31.754596 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:31.754598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:31.754739 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:31.755173 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:31.755176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:31.755178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:31.755646 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:31.755798 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:31.756367 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 20:58:31.756667 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":455,"thread_start_us":299,"threads_started":1}
I20260929 20:58:31.757736 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:31.738841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43927 (local address 127.0.0.1:39128)
0929 20:58:31.739643 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:31.739691 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:58:31.741673 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:58:31.742587 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:58:31.742592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:58:31.743145 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:58:31.743509 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:31.743527 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:31.745496 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:31.745500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:31.746603 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:31.746615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:31.747770 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:31.747775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:31.747935 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:31.748590 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:58:31.748611 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:58:31.750924 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:58:31.753954 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:31.753964 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:31.753974 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:31.754450 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:31.754844 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:31.754849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:31.754853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:31.755073 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:31.755775 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:58:31.755784 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:58:31.756012 (+ 228us) client_negotiation.cc:769] Sending connection context
0929 20:58:31.756264 (+ 252us) client_negotiation.cc:240] Negotiation successful
0929 20:58:31.756814 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":98,"thread_start_us":323,"threads_started":1}
I20260929 20:58:31.760579 22796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:58:31.761616 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:31.761252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34009 (local address 127.0.0.1:60662)
0929 20:58:31.761398 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:31.761492 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 20:58:31.761917 22745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34009: connect: Connection refused (error 111)
I20260929 20:58:31.763307 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:31.763008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34009 (local address 127.0.0.1:60674)
0929 20:58:31.763131 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:31.763195 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 20:58:31.763492 22796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34009: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34009: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 20:58:31.783073 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34009
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43927,127.0.97.61:34009,127.0.97.60:35711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:58:31.812814 22796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } attempt: 1
I20260929 20:58:31.813944 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:31.813528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34009 (local address 127.0.0.1:60686)
0929 20:58:31.813717 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:31.813806 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 20:58:31.814952 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:31.814588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34009 (local address 127.0.0.1:60702)
0929 20:58:31.814745 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:31.814818 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 20:58:31.815143 22796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34009: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34009: connect: Connection refused (error 111)
I20260929 20:58:31.883381 22796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } attempt: 2
I20260929 20:58:31.884568 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:31.884080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34009 (local address 127.0.0.1:60710)
0929 20:58:31.884302 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:31.884399 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 20:58:31.885484 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:31.885155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34009 (local address 127.0.0.1:60720)
0929 20:58:31.885286 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:31.885371 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 20:58:31.885654 22796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34009: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34009: connect: Connection refused (error 111)
I20260929 20:58:31.976886 22796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } attempt: 3
I20260929 20:58:31.977901 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:31.977498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34009 (local address 127.0.0.1:60736)
0929 20:58:31.977685 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:31.977772 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 20:58:31.978852 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:31.978518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34009 (local address 127.0.0.1:60744)
0929 20:58:31.978657 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:31.978728 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 20:58:31.979022 22796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34009: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34009: connect: Connection refused (error 111)
W20260929 20:58:32.103981 22803 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:58:32.104332 22803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:32.104430 22803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:32.104542 22803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:32.111853 22803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:32.111966 22803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:32.112047 22803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:32.112111 22803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:32.112197 22803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:32.122103 22803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43927,127.0.97.61:34009,127.0.97.60:35711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:32.122274 22796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } attempt: 4
I20260929 20:58:32.123405 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.122879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34009 (local address 127.0.0.1:60760)
0929 20:58:32.123188 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.123278 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260929 20:58:32.123492 22803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:32.124533 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.124055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34009 (local address 127.0.0.1:60774)
0929 20:58:32.124236 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.124341 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 20:58:32.124763 22796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34009: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34009: connect: Connection refused (error 111)
I20260929 20:58:32.125067 22803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:32.136297 22808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:32.136261 22809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:32.136447 22811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:32.136874 22803 server_base.cc:1034] running on GCE node
I20260929 20:58:32.137615 22803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:32.139180 22803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:32.140453 22803 hybrid_clock.cc:648] HybridClock initialized: now 1790715512140378 us; error 42 us; skew 500 ppm
I20260929 20:58:32.140799 22803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:32 dist-test-slave-m9g1 krb5kdc[22716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:32.144059 22803 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 20:58:32.145601 22803 webserver.cc:460] Webserver started at http://127.0.97.61:34527/ using document root <none> and password file <none>
I20260929 20:58:32.146348 22803 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:32.146450 22803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:32.146752 22803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:32.149708 22803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-1/data/instance:
uuid: "d088e79039e64a75a2aacc8313025927"
format_stamp: "Formatted at 2026-09-29 20:58:32 on dist-test-slave-m9g1"
server_key: "6ad0ff3dfb7553efb60653543063e579"
server_key_iv: "090e725fc9811bc4c04d1eb3b935a9bf"
server_key_version: "encryptionkey@0"
I20260929 20:58:32.150504 22803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-1/wal/instance:
uuid: "d088e79039e64a75a2aacc8313025927"
format_stamp: "Formatted at 2026-09-29 20:58:32 on dist-test-slave-m9g1"
server_key: "6ad0ff3dfb7553efb60653543063e579"
server_key_iv: "090e725fc9811bc4c04d1eb3b935a9bf"
server_key_version: "encryptionkey@0"
I20260929 20:58:32.155544 22803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 20:58:32.159162 22818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:32.160602 22803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:58:32.160771 22803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-1/wal
uuid: "d088e79039e64a75a2aacc8313025927"
format_stamp: "Formatted at 2026-09-29 20:58:32 on dist-test-slave-m9g1"
server_key: "6ad0ff3dfb7553efb60653543063e579"
server_key_iv: "090e725fc9811bc4c04d1eb3b935a9bf"
server_key_version: "encryptionkey@0"
I20260929 20:58:32.160995 22803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:32.178645 22803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:32.182950 22803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:32.183192 22803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:32.200003 22803 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34009
I20260929 20:58:32.200037 22869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34009 every 8 connection(s)
I20260929 20:58:32.201283 22803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-1/data/info.pb
I20260929 20:58:32.204846 22870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:32.205974 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 22803
I20260929 20:58:32.206146 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-1/wal/instance
I20260929 20:58:32.206583 388 external_mini_cluster.cc:1428] Setting key 40fad517d15f79c59c2c797e1a49cf53
I20260929 20:58:32.211355 22870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:58:32 dist-test-slave-m9g1 krb5kdc[22716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260929 20:58:32.231122 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.213933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39140 (local address 127.0.97.62:43927)
0929 20:58:32.214140 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:32.214147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:32.215292 (+ 1145us) server_negotiation.cc:407] Connection header received
0929 20:58:32.217129 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:32.217135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:32.217213 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:32.217323 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:32.218778 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.219719 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.221150 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.221726 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.221851 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.225284 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:32.225291 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:32.225297 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:32.225339 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:32.228488 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.229126 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.229130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.229132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.229248 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.229592 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.229597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.229600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.229879 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:32.229988 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:32.230729 (+ 741us) server_negotiation.cc:299] Negotiation successful
0929 20:58:32.230900 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:58:32.232594 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.214356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43927 (local address 127.0.0.1:39140)
0929 20:58:32.215133 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.215175 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:58:32.216881 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:58:32.217521 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:58:32.217529 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:58:32.218023 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:58:32.218621 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:32.218639 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.219853 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.219857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:32.221013 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:32.221024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.221907 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.221911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:32.222101 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.222778 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:58:32.222800 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:58:32.225048 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:58:32.228613 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:32.228620 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:32.228633 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:32.229026 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:32.229324 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:32.229327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:32.229329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:32.229491 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:32.229967 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:58:32.229973 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:58:32.230354 (+ 381us) client_negotiation.cc:769] Sending connection context
0929 20:58:32.230626 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 20:58:32.231112 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":327,"threads_started":1}
I20260929 20:58:32.234320 22870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:58:32 dist-test-slave-m9g1 krb5kdc[22716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:58:32.248741 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.235019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34009 (local address 127.0.0.1:60776)
0929 20:58:32.235214 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.235232 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:58:32.235326 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:58:32.236274 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:58:32.236278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:58:32.236633 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:58:32.236769 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:32.236778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.238544 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.238548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:32.239356 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:32.239366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.240441 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.240446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:32.240564 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.241102 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:58:32.241118 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:58:32.243124 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:58:32.246632 (+ 3508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:32.246636 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:32.246639 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:32.246973 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:32.247321 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:32.247324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:32.247326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:32.247453 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:32.248066 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:58:32.248073 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:58:32.248192 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 20:58:32.248294 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 20:58:32.248485 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 20:58:32.249058 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.235238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60776 (local address 127.0.97.61:34009)
0929 20:58:32.235830 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:32.235837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:32.235864 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:58:32.235909 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:32.235916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:32.236135 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:32.236262 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:32.236926 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.238408 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.239498 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.240305 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.240435 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.243323 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:32.243335 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:32.243350 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:32.243387 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:32.246522 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.247089 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.247096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.247099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.247233 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.247556 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.247561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.247564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.247951 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:32.248156 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:32.248437 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 20:58:32.248674 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":312,"threads_started":1}
I20260929 20:58:32.251379 22870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:58:32.252595 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.252088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:40934)
0929 20:58:32.252266 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.252399 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 20:58:32.252844 22820 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
I20260929 20:58:32.254369 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.253980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:40936)
0929 20:58:32.254137 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.254244 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 20:58:32.254598 22870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 20:58:32.263203 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35711
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43927,127.0.97.61:34009,127.0.97.60:35711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:58:32.303943 22870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } attempt: 1
I20260929 20:58:32.305181 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.304647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:40948)
0929 20:58:32.304874 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.304992 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 20:58:32.306180 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.305789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:40962)
0929 20:58:32.305953 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.306054 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 20:58:32.306427 22870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
I20260929 20:58:32.374675 22870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } attempt: 2
I20260929 20:58:32.375845 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.375378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:40974)
0929 20:58:32.375596 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.375690 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 20:58:32.376823 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.376520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:40976)
0929 20:58:32.376641 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.376709 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 20:58:32.377033 22870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
I20260929 20:58:32.424031 22796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } attempt: 5
Sep 29 20:58:32 dist-test-slave-m9g1 krb5kdc[22716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:58:32.438799 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.424692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34009 (local address 127.0.0.1:60780)
0929 20:58:32.424920 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.424943 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:58:32.425075 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:58:32.425431 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:58:32.425437 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:58:32.425932 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:58:32.426156 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:32.426170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.427237 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.427242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:32.428613 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:32.428628 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.429358 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.429364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:32.429586 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.430257 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:58:32.430279 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:58:32.432712 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:58:32.436470 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:32.436476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:32.436480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:32.436882 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:32.437254 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:32.437258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:32.437260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:32.437417 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:32.438003 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:58:32.438007 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:58:32.438144 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 20:58:32.438334 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 20:58:32.438548 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 20:58:32.438949 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.424866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60780 (local address 127.0.97.61:34009)
0929 20:58:32.425030 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:32.425035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:32.425055 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:58:32.425197 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:32.425200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:32.425276 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:32.425366 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:32.426394 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.427080 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.428771 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.429202 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.429328 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.432935 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:32.432942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:32.432947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:32.432992 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:32.436299 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.436992 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.436997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.437001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.437143 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.437512 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.437516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.437520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.437874 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:32.438015 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:32.438421 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 20:58:32.438654 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:58:32.440026 22796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:58:32.441151 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.440764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:40980)
0929 20:58:32.440929 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.441008 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 20:58:32.442091 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.441747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:40984)
0929 20:58:32.441898 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.441996 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 20:58:32.442312 22796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
I20260929 20:58:32.468231 22870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } attempt: 3
I20260929 20:58:32.469240 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.468790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:40990)
0929 20:58:32.468996 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.469084 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 20:58:32.470291 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.469833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:40994)
0929 20:58:32.470054 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.470129 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260929 20:58:32.470494 22870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
I20260929 20:58:32.493537 22796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } attempt: 1
I20260929 20:58:32.494583 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.494126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:41002)
0929 20:58:32.494331 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.494427 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 20:58:32.495558 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.495236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:41010)
0929 20:58:32.495373 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.495458 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 20:58:32.495752 22796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
I20260929 20:58:32.563978 22796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } attempt: 2
I20260929 20:58:32.565223 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.564718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:41022)
0929 20:58:32.564945 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.565046 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 20:58:32.566255 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.565861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:41036)
0929 20:58:32.566025 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.566113 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 20:58:32.566439 22796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
W20260929 20:58:32.590898 22877 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:58:32.591259 22877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:32.591372 22877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:32.591491 22877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:32.598892 22877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:32.599000 22877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:32.599076 22877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:32.599139 22877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:32.599222 22877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:32.609102 22877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43927,127.0.97.61:34009,127.0.97.60:35711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:32.610430 22877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:32.611732 22877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 20:58:32.613688 22870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } attempt: 4
I20260929 20:58:32.614837 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.614388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:41044)
0929 20:58:32.614610 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.614692 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 20:58:32.615813 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.615499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:41046)
0929 20:58:32.615640 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.615709 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 20:58:32.615986 22870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
W20260929 20:58:32.623055 22882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:32.622990 22885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:32.623029 22883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:32.623768 22877 server_base.cc:1034] running on GCE node
I20260929 20:58:32.624485 22877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:32.625828 22877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:32.627044 22877 hybrid_clock.cc:648] HybridClock initialized: now 1790715512626955 us; error 47 us; skew 500 ppm
I20260929 20:58:32.627337 22877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:32 dist-test-slave-m9g1 krb5kdc[22716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:32.630421 22877 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 20:58:32.631836 22877 webserver.cc:460] Webserver started at http://127.0.97.60:35415/ using document root <none> and password file <none>
I20260929 20:58:32.632499 22877 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:32.632589 22877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:32.632880 22877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:32.635818 22877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-2/data/instance:
uuid: "d620b93aea4e49e7b8d78042bcc15b92"
format_stamp: "Formatted at 2026-09-29 20:58:32 on dist-test-slave-m9g1"
server_key: "b76f5c40db847be6b776decf26147405"
server_key_iv: "bff5bd1f22e3bccbda32329f5b46a00d"
server_key_version: "encryptionkey@0"
I20260929 20:58:32.636581 22877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-2/wal/instance:
uuid: "d620b93aea4e49e7b8d78042bcc15b92"
format_stamp: "Formatted at 2026-09-29 20:58:32 on dist-test-slave-m9g1"
server_key: "b76f5c40db847be6b776decf26147405"
server_key_iv: "bff5bd1f22e3bccbda32329f5b46a00d"
server_key_version: "encryptionkey@0"
I20260929 20:58:32.641557 22877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 20:58:32.644896 22892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:32.646482 22877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:58:32.646661 22877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-2/wal
uuid: "d620b93aea4e49e7b8d78042bcc15b92"
format_stamp: "Formatted at 2026-09-29 20:58:32 on dist-test-slave-m9g1"
server_key: "b76f5c40db847be6b776decf26147405"
server_key_iv: "bff5bd1f22e3bccbda32329f5b46a00d"
server_key_version: "encryptionkey@0"
I20260929 20:58:32.646883 22877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:32.663065 22877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:32.667833 22877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:32.668090 22877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:32.672650 22796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } attempt: 3
I20260929 20:58:32.673923 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.673316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:41054)
0929 20:58:32.673605 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.673720 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 20:58:32.675218 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.674745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:41056)
0929 20:58:32.674955 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.675068 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260929 20:58:32.675453 22796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35711: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35711: connect: Connection refused (error 111)
I20260929 20:58:32.686888 22877 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:35711
I20260929 20:58:32.686920 22943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:35711 every 8 connection(s)
I20260929 20:58:32.688215 22877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-2/data/info.pb
I20260929 20:58:32.692021 22944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:32.696702 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 22877
I20260929 20:58:32.696842 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715169435170-388-0/minicluster-data/master-2/wal/instance
I20260929 20:58:32.697273 388 external_mini_cluster.cc:1428] Setting key 9d45766af1ae51cc9d5cf4e50c3e5e2f
I20260929 20:58:32.698539 22944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:58:32 dist-test-slave-m9g1 krb5kdc[22716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 29 20:58:32 dist-test-slave-m9g1 krb5kdc[22716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 20:58:32.716519 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.701262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39144 (local address 127.0.97.62:43927)
0929 20:58:32.701491 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:32.701497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:32.702020 (+ 523us) server_negotiation.cc:407] Connection header received
0929 20:58:32.702508 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:32.702514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:32.702590 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:32.702691 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:32.703870 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.704555 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.706130 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.706736 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.706822 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.710358 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:32.710369 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:32.710373 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:32.710408 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:32.713100 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.714010 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.714014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.714016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.714144 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.714615 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.714621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.714624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.714979 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:32.715118 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:32.715925 (+ 807us) server_negotiation.cc:299] Negotiation successful
0929 20:58:32.716134 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:58:32.717832 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.701200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43927 (local address 127.0.0.1:39144)
0929 20:58:32.701870 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.701916 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:58:32.702235 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:58:32.702767 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:58:32.702774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:58:32.703377 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:58:32.703714 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:32.703730 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.704701 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.704707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:32.705990 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:32.706010 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.706941 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.706948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:32.707217 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.707924 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:58:32.707954 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:58:32.710119 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:58:32.713228 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:32.713240 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:32.713258 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:32.713889 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:32.714273 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:32.714278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:32.714281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:32.714514 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:32.715084 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:58:32.715092 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:58:32.715509 (+ 417us) client_negotiation.cc:769] Sending connection context
0929 20:58:32.715842 (+ 333us) client_negotiation.cc:240] Negotiation successful
0929 20:58:32.716351 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":283,"threads_started":1}
I20260929 20:58:32.717850 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.699080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.0.97.60:35711)
0929 20:58:32.699885 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:32.699896 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:32.699938 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:58:32.701812 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:32.701843 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:32.702261 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:32.702530 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:32.703857 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.705135 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.706584 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.707230 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.707346 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.711067 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:32.711076 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:32.711088 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:32.711116 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:32.714321 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.715062 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.715070 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.715074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.715221 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.715548 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.715552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.715555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.715972 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:32.716247 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:32.716555 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 20:58:32.716771 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":380,"threads_started":1}
I20260929 20:58:32.720844 22944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:58:32 dist-test-slave-m9g1 krb5kdc[22716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:58:32.734429 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.721531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34009 (local address 127.0.0.1:60782)
0929 20:58:32.721788 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.721802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:58:32.721885 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:58:32.722144 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:58:32.722146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:58:32.722504 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:58:32.722645 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:32.722652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.723992 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.723996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:32.724847 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:32.724857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.725886 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.725890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:32.726034 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.726606 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:58:32.726622 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:58:32.728713 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:58:32.732172 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:32.732178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:32.732182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:32.732538 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:32.732891 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:32.732897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:32.732901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:32.733043 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:32.733512 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:58:32.733517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:58:32.733656 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 20:58:32.733771 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 20:58:32.734055 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260929 20:58:32.734460 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.721533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60782 (local address 127.0.97.61:34009)
0929 20:58:32.721688 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:32.721692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:32.721882 (+ 190us) server_negotiation.cc:407] Connection header received
0929 20:58:32.721962 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:32.721968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:32.722052 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:32.722139 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:32.722763 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.723839 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.724984 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.725739 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.725864 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.729007 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:32.729012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:32.729016 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:32.729051 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:32.732056 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.732650 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.732654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.732657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.732791 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.733143 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.733147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.733149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.733391 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:32.733534 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:32.733874 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 20:58:32.734073 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 20:58:32.735486 22944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:58:32 dist-test-slave-m9g1 krb5kdc[22716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:58:32.741853 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.720902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39148 (local address 127.0.97.62:43927)
0929 20:58:32.728252 (+ 7350us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:32.728259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:32.728278 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:58:32.728332 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:32.728338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:32.728411 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:32.728530 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:32.730078 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.731178 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.732338 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.732752 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.732858 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.736742 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:32.736749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:32.736754 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:32.736791 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:32.739425 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.740022 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.740027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.740031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.740170 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.740556 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.740561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.740563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.740831 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:32.741091 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:32.741359 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 20:58:32.741583 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7210}
Sep 29 20:58:32 dist-test-slave-m9g1 krb5kdc[22716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 20:58:32.747902 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.736201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:41076)
0929 20:58:32.736374 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.736394 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:58:32.736498 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:58:32.736770 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:58:32.736773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:58:32.737099 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:58:32.737232 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:32.737243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.738501 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.738505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:32.739321 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:32.739331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.740038 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.740042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:32.740210 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.740723 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:58:32.740739 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:58:32.742713 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:58:32.745662 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:32.745666 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:32.745668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:32.746031 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:32.746400 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:32.746405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:32.746409 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:32.746530 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:32.747006 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:58:32.747010 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:58:32.747130 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 20:58:32.747248 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 20:58:32.747511 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 20:58:32.747938 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.736351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.0.97.60:35711)
0929 20:58:32.736482 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:32.736486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:32.736502 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:58:32.736593 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:32.736596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:32.736655 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:32.736743 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:32.737362 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.738390 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.739448 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.739900 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.740011 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.742881 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:32.742885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:32.742888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:32.742915 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:32.745541 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.746138 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.746143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.746146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.746308 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.746631 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.746636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.746640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.746899 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:32.747018 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:32.747336 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 20:58:32.747662 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":41}
Sep 29 20:58:32 dist-test-slave-m9g1 krb5kdc[22716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:58:32.753261 22944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92: Bootstrap starting.
I20260929 20:58:32.756316 22944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:32.756736 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.743418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60788 (local address 127.0.97.61:34009)
0929 20:58:32.743598 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:32.743604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:32.743622 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:58:32.743676 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:32.743681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:32.743751 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:32.743851 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:32.745224 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.746105 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.747327 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.747744 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.747864 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.751866 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:32.751872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:32.751875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:32.751906 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:32.754359 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.754946 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.754952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.754956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.755079 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.755466 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.755471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.755475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.755732 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:32.755985 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:32.756218 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 20:58:32.756439 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:58:32.757412 22944 log.cc:824] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:32.760596 22944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92: No bootstrap required, opened a new log
I20260929 20:58:32.761318 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:58:32.764901 22771 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "15538a1d52cb44389c15aa2e5a8306eb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39148
I20260929 20:58:32.771227 22944 raft_consensus.cc:348] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538a1d52cb44389c15aa2e5a8306eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } } peers { permanent_uuid: "d088e79039e64a75a2aacc8313025927" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } } peers { permanent_uuid: "d620b93aea4e49e7b8d78042bcc15b92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } }
I20260929 20:58:32.771688 22944 raft_consensus.cc:729] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d620b93aea4e49e7b8d78042bcc15b92, State: Initialized, Role: FOLLOWER
I20260929 20:58:32.772158 22944 consensus_queue.cc:260] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538a1d52cb44389c15aa2e5a8306eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } } peers { permanent_uuid: "d088e79039e64a75a2aacc8313025927" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } } peers { permanent_uuid: "d620b93aea4e49e7b8d78042bcc15b92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } }
I20260929 20:58:32.773139 22950 sys_catalog.cc:455] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538a1d52cb44389c15aa2e5a8306eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } } peers { permanent_uuid: "d088e79039e64a75a2aacc8313025927" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } } peers { permanent_uuid: "d620b93aea4e49e7b8d78042bcc15b92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } } }
I20260929 20:58:32.773517 22950 sys_catalog.cc:458] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:58:32.774111 22944 sys_catalog.cc:564] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 20:58:32.782975 22961 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:58:32.783106 22961 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:58:32.781085 22877 master_runner.cc:186] Error getting master registration for 127.0.97.62:43927: OK, instance_id { permanent_uuid: "15538a1d52cb44389c15aa2e5a8306eb" instance_seqno: 1790715511716387 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:58:32.792191 22877 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:58:32.852643 22796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } attempt: 4
Sep 29 20:58:32 dist-test-slave-m9g1 krb5kdc[22716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 20:58:32.867523 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.853398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:41080)
0929 20:58:32.853599 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.853619 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:58:32.853755 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:58:32.854105 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:58:32.854110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:58:32.854548 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:58:32.854717 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:32.854728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.855746 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.855752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:32.856688 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:32.856701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.857447 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.857452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:32.857656 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.858270 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:58:32.858287 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:58:32.860611 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:58:32.864673 (+ 4062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:32.864677 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:32.864680 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:32.865081 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:32.865472 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:32.865477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:32.865481 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:32.865689 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:32.866459 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:58:32.866464 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:58:32.866657 (+ 193us) client_negotiation.cc:769] Sending connection context
0929 20:58:32.866808 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 20:58:32.867122 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 20:58:32.867579 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.853460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.0.97.60:35711)
0929 20:58:32.853688 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:32.853694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:32.853716 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:58:32.853884 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:32.853889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:32.853956 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:32.854040 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:32.854873 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.855601 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.856830 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.857301 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.857408 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.860829 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:32.860837 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:32.860843 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:32.860888 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:32.864486 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.865194 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.865200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.865204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.865370 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.865811 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.865817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.865820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.866347 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:32.866515 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:32.866933 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 20:58:32.867190 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 20:58:32.873430 22796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb: Bootstrap starting.
I20260929 20:58:32.876906 22796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:32.877951 22796 log.cc:824] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:32.880585 22796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb: No bootstrap required, opened a new log
I20260929 20:58:32.885913 22796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538a1d52cb44389c15aa2e5a8306eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } } peers { permanent_uuid: "d088e79039e64a75a2aacc8313025927" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } } peers { permanent_uuid: "d620b93aea4e49e7b8d78042bcc15b92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } }
I20260929 20:58:32.886263 22796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15538a1d52cb44389c15aa2e5a8306eb, State: Initialized, Role: FOLLOWER
I20260929 20:58:32.886682 22796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538a1d52cb44389c15aa2e5a8306eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } } peers { permanent_uuid: "d088e79039e64a75a2aacc8313025927" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } } peers { permanent_uuid: "d620b93aea4e49e7b8d78042bcc15b92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } }
I20260929 20:58:32.887569 22964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538a1d52cb44389c15aa2e5a8306eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } } peers { permanent_uuid: "d088e79039e64a75a2aacc8313025927" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } } peers { permanent_uuid: "d620b93aea4e49e7b8d78042bcc15b92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } } }
I20260929 20:58:32.887914 22964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:58:32.888653 22796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:32.888789 22771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 20:58:32.889050 22771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:32.892731 22771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538a1d52cb44389c15aa2e5a8306eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } } peers { permanent_uuid: "d088e79039e64a75a2aacc8313025927" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } } peers { permanent_uuid: "d620b93aea4e49e7b8d78042bcc15b92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } }
I20260929 20:58:32.896178 22771 leader_election.cc:288] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [CANDIDATE]: Term 1 election: Requested vote from peers d088e79039e64a75a2aacc8313025927 (127.0.97.61:34009), d620b93aea4e49e7b8d78042bcc15b92 (127.0.97.60:35711)
I20260929 20:58:32.896562 22919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15538a1d52cb44389c15aa2e5a8306eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d620b93aea4e49e7b8d78042bcc15b92"
I20260929 20:58:32.897128 22919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:32.900632 22919 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15538a1d52cb44389c15aa2e5a8306eb in term 1.
I20260929 20:58:32.901372 22746 leader_election.cc:302] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15538a1d52cb44389c15aa2e5a8306eb, d620b93aea4e49e7b8d78042bcc15b92; no voters:
I20260929 20:58:32.901856 22964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [term 1 FOLLOWER]: Leader election won for term 1
W20260929 20:58:32.906555 22975 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:58:32.906728 22975 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:58:32.911322 22964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [term 1 LEADER]: Becoming Leader. State: Replica: 15538a1d52cb44389c15aa2e5a8306eb, State: Running, Role: LEADER
I20260929 20:58:32.912035 22964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538a1d52cb44389c15aa2e5a8306eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } } peers { permanent_uuid: "d088e79039e64a75a2aacc8313025927" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } } peers { permanent_uuid: "d620b93aea4e49e7b8d78042bcc15b92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } }
I20260929 20:58:32.915161 22870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } attempt: 5
W20260929 20:58:32.920137 22898 tablet.cc:2369] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 20:58:32 dist-test-slave-m9g1 krb5kdc[22716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 20:58:32.899737 22729 master_runner.cc:186] Error getting master registration for 127.0.97.61:34009: OK, instance_id { permanent_uuid: "d088e79039e64a75a2aacc8313025927" instance_seqno: 1790715512195106 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:58:32.948015 22729 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:58:32.951512 22979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15538a1d52cb44389c15aa2e5a8306eb. Latest consensus state: current_term: 1 leader_uuid: "15538a1d52cb44389c15aa2e5a8306eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538a1d52cb44389c15aa2e5a8306eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } } peers { permanent_uuid: "d088e79039e64a75a2aacc8313025927" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } } peers { permanent_uuid: "d620b93aea4e49e7b8d78042bcc15b92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } } }
I20260929 20:58:32.951936 22979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:32.952780 22987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:58:32.954919 22987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260929 20:58:32.958422 22750 tablet.cc:2369] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:58:32.964154 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.915694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35711 (local address 127.0.0.1:41106)
0929 20:58:32.916039 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:32.916059 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:58:32.916179 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:58:32.933755 (+ 17576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:58:32.933761 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:58:32.934287 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:58:32.934529 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:32.934544 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.939287 (+ 4743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.939294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:32.940680 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:32.940701 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.941708 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.941714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:32.941933 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.942658 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:58:32.942684 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:58:32.945390 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:58:32.957200 (+ 11810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:32.957205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:32.957209 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:32.957783 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:32.958276 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:32.958281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:32.958285 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:32.958475 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:32.960841 (+ 2366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:58:32.960846 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:58:32.961021 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 20:58:32.961170 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 20:58:32.961489 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229}
I20260929 20:58:32.965376 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.916397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41106 (local address 127.0.97.60:35711)
0929 20:58:32.933240 (+ 16843us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:32.933247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:32.933272 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:58:32.933357 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:32.933363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:32.933455 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:32.934692 (+ 1237us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:32.934952 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.939119 (+ 4167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.940890 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.941527 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.945564 (+ 4037us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.945819 (+ 255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:32.945826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:32.945831 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:32.945863 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:32.957027 (+ 11164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.957955 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.957960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.957964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.958116 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.958620 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.958624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.958627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.958947 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:32.964583 (+ 5636us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:32.964730 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 20:58:32.964962 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16701,"thread_start_us":17145,"threads_started":1}
I20260929 20:58:32.971655 22870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927: Bootstrap starting.
I20260929 20:58:32.975924 22870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:32.979652 22870 log.cc:824] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:32.989943 22870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927: No bootstrap required, opened a new log
I20260929 20:58:32.993932 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.910251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.0.97.60:35711)
0929 20:58:32.910438 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:32.910444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:32.933477 (+ 23033us) server_negotiation.cc:407] Connection header received
0929 20:58:32.935079 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:32.935083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:32.935171 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:32.935233 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:32.952169 (+ 16936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.953242 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.965675 (+ 12433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.966352 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.966442 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.977921 (+ 11479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:32.977928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:32.977934 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:32.977978 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:32.982699 (+ 4721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.989101 (+ 6402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.989109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.989113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.989309 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.989786 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.989793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.989797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.990175 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:32.990502 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:32.993237 (+ 2735us) server_negotiation.cc:299] Negotiation successful
0929 20:58:32.993536 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:58:32.998929 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.909576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.0.97.62:43927)
0929 20:58:32.909752 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:32.909758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:32.951789 (+ 42031us) server_negotiation.cc:407] Connection header received
0929 20:58:32.954301 (+ 2512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:32.954307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:32.954393 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:32.954479 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:32.969443 (+ 14964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.970500 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.984980 (+ 14480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.985726 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.985874 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:32.990865 (+ 4991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:32.990874 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:32.990878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:32.990924 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:32.994612 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.995840 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.995847 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.995850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.996026 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:32.996624 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:32.996629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:32.996633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:32.997048 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:32.997302 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:32.998112 (+ 810us) server_negotiation.cc:299] Negotiation successful
0929 20:58:32.998394 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:58:33.007354 22919 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92 [term 1 FOLLOWER]: Refusing update from remote peer 15538a1d52cb44389c15aa2e5a8306eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:58:33.008359 22964 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d620b93aea4e49e7b8d78042bcc15b92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:58:33.012964 22870 raft_consensus.cc:348] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538a1d52cb44389c15aa2e5a8306eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } } peers { permanent_uuid: "d088e79039e64a75a2aacc8313025927" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } } peers { permanent_uuid: "d620b93aea4e49e7b8d78042bcc15b92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } }
I20260929 20:58:33.013386 22870 raft_consensus.cc:729] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d088e79039e64a75a2aacc8313025927, State: Initialized, Role: FOLLOWER
I20260929 20:58:33.013955 22870 consensus_queue.cc:260] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538a1d52cb44389c15aa2e5a8306eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } } peers { permanent_uuid: "d088e79039e64a75a2aacc8313025927" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } } peers { permanent_uuid: "d620b93aea4e49e7b8d78042bcc15b92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } }
I20260929 20:58:33.017589 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:32.911330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60800 (local address 127.0.97.61:34009)
0929 20:58:32.911501 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:32.911507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:32.969226 (+ 57719us) server_negotiation.cc:407] Connection header received
0929 20:58:32.971634 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:32.971645 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:32.971737 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:32.971825 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:32.985038 (+ 13213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.986220 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.991024 (+ 4804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:32.991769 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:32.991876 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:33.006853 (+ 14977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:33.006861 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:33.006866 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:33.006916 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:33.010983 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:33.012444 (+ 1461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:33.012451 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:33.012455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:33.012706 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:33.013416 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:33.013422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:33.013425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:33.014016 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:33.014814 (+ 798us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:33.015428 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 20:58:33.016318 (+ 890us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:58:33.022787 22870 sys_catalog.cc:564] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:33.023546 22845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15538a1d52cb44389c15aa2e5a8306eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d088e79039e64a75a2aacc8313025927"
I20260929 20:58:33.024883 22844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:33.028662 22844 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927 [term 1 FOLLOWER]: Refusing update from remote peer 15538a1d52cb44389c15aa2e5a8306eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:58:33.029740 22964 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d088e79039e64a75a2aacc8313025927" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:58:33.033005 22845 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15538a1d52cb44389c15aa2e5a8306eb in term 1.
I20260929 20:58:33.058694 22950 sys_catalog.cc:455] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15538a1d52cb44389c15aa2e5a8306eb. Latest consensus state: current_term: 1 leader_uuid: "15538a1d52cb44389c15aa2e5a8306eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538a1d52cb44389c15aa2e5a8306eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } } peers { permanent_uuid: "d088e79039e64a75a2aacc8313025927" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } } peers { permanent_uuid: "d620b93aea4e49e7b8d78042bcc15b92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } } }
I20260929 20:58:33.059077 22950 sys_catalog.cc:458] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:58:33.069458 22950 sys_catalog.cc:455] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15538a1d52cb44389c15aa2e5a8306eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538a1d52cb44389c15aa2e5a8306eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } } peers { permanent_uuid: "d088e79039e64a75a2aacc8313025927" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } } peers { permanent_uuid: "d620b93aea4e49e7b8d78042bcc15b92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } } }
I20260929 20:58:33.069774 22950 sys_catalog.cc:458] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:58:33.073328 22979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15538a1d52cb44389c15aa2e5a8306eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538a1d52cb44389c15aa2e5a8306eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } } peers { permanent_uuid: "d088e79039e64a75a2aacc8313025927" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } } peers { permanent_uuid: "d620b93aea4e49e7b8d78042bcc15b92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } } }
I20260929 20:58:33.073632 22979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:33.075264 22991 sys_catalog.cc:455] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15538a1d52cb44389c15aa2e5a8306eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538a1d52cb44389c15aa2e5a8306eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } } peers { permanent_uuid: "d088e79039e64a75a2aacc8313025927" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } } peers { permanent_uuid: "d620b93aea4e49e7b8d78042bcc15b92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } } }
I20260929 20:58:33.075791 22991 sys_catalog.cc:458] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:58:33.076038 22987 catalog_manager.cc:1347] Generated new cluster ID: c93f51ab6c3b406eaf1c573994c30981
I20260929 20:58:33.076133 22987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:58:33.079154 22964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15538a1d52cb44389c15aa2e5a8306eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538a1d52cb44389c15aa2e5a8306eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } } peers { permanent_uuid: "d088e79039e64a75a2aacc8313025927" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } } peers { permanent_uuid: "d620b93aea4e49e7b8d78042bcc15b92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } } }
I20260929 20:58:33.079531 22964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:33.082077 22994 sys_catalog.cc:455] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15538a1d52cb44389c15aa2e5a8306eb. Latest consensus state: current_term: 1 leader_uuid: "15538a1d52cb44389c15aa2e5a8306eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538a1d52cb44389c15aa2e5a8306eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } } peers { permanent_uuid: "d088e79039e64a75a2aacc8313025927" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } } peers { permanent_uuid: "d620b93aea4e49e7b8d78042bcc15b92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } } }
I20260929 20:58:33.086505 22994 sys_catalog.cc:458] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:58:33.110062 22994 sys_catalog.cc:455] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15538a1d52cb44389c15aa2e5a8306eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538a1d52cb44389c15aa2e5a8306eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43927 } } peers { permanent_uuid: "d088e79039e64a75a2aacc8313025927" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34009 } } peers { permanent_uuid: "d620b93aea4e49e7b8d78042bcc15b92" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35711 } } }
I20260929 20:58:33.110459 22994 sys_catalog.cc:458] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:58:33.157595 22987 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:58:33.159794 22987 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:58:33.175554 22987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15538a1d52cb44389c15aa2e5a8306eb: Generated new TSK 0
I20260929 20:58:33.179190 22987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:58:33.205375 23010 catalog_manager.cc:1259] Loaded cluster ID: c93f51ab6c3b406eaf1c573994c30981
I20260929 20:58:33.205492 23010 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927: loading cluster ID for follower catalog manager: success
I20260929 20:58:33.209084 23010 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927: acquiring CA information for follower catalog manager: success
I20260929 20:58:33.211491 23010 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d088e79039e64a75a2aacc8313025927: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 20:58:33.320458 22874 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:33.295035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60816 (local address 127.0.97.61:34009)
0929 20:58:33.295248 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.295256 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.317003 (+ 21747us) server_negotiation.cc:407] Connection header received
0929 20:58:33.318898 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.318905 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.319019 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.319094 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:33.320222 (+ 1128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60816: BlockingRecv error: recv got EOF from 127.0.0.1:60816 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 20:58:33.322641 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:33.301607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41118 (local address 127.0.97.60:35711)
0929 20:58:33.301792 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.301799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.318602 (+ 16803us) server_negotiation.cc:407] Connection header received
0929 20:58:33.321145 (+ 2543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.321151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.321233 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.321303 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:33.322356 (+ 1053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41118: BlockingRecv error: recv got EOF from 127.0.0.1:41118 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 20:58:33.325286 22800 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:33.314704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39160 (local address 127.0.97.62:43927)
0929 20:58:33.314920 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.314927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.323488 (+ 8561us) server_negotiation.cc:407] Connection header received
0929 20:58:33.323873 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.323879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.323975 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.324047 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:33.325057 (+ 1010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39160: BlockingRecv error: recv got EOF from 127.0.0.1:39160 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:58:33.376677 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:33.357236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.0.97.61:34009)
0929 20:58:33.357426 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.357432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.368245 (+ 10813us) server_negotiation.cc:407] Connection header received
0929 20:58:33.368842 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.368848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.369133 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.369282 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:58:33.370056 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.371409 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.373455 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.374133 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.374255 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:33.375081 (+ 826us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:58:33.375832 (+ 751us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:58:33.375957 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:33.376242 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 20:58:33.376344 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 20:58:33.395411 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:33.393221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.0.97.60:35711)
0929 20:58:33.393402 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.393409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.393918 (+ 509us) server_negotiation.cc:407] Connection header received
0929 20:58:33.394141 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.394147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.394310 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.394421 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:33.395197 (+ 776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41130: BlockingRecv error: recv got EOF from 127.0.0.1:41130 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:58:33.408773 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:33.399899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60824 (local address 127.0.97.61:34009)
0929 20:58:33.400067 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.400073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.402719 (+ 2646us) server_negotiation.cc:407] Connection header received
0929 20:58:33.402882 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.402888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.403002 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.403108 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:58:33.403570 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.404717 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.406687 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.407375 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.407511 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:33.407962 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:58:33.408133 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:58:33.408226 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:33.408364 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 20:58:33.408456 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 20:58:33.422833 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:33.420577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41142 (local address 127.0.97.60:35711)
0929 20:58:33.420770 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.420777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.421329 (+ 552us) server_negotiation.cc:407] Connection header received
0929 20:58:33.421481 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.421488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.421610 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.421720 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:33.422589 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41142: BlockingRecv error: recv got EOF from 127.0.0.1:41142 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:58:33.435459 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:33.428335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60836 (local address 127.0.97.61:34009)
0929 20:58:33.428534 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.428540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.429022 (+ 482us) server_negotiation.cc:407] Connection header received
0929 20:58:33.429214 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.429220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.429425 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.429533 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:58:33.430104 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.431314 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.433281 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.433948 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.434072 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:33.434563 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:58:33.434710 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:58:33.434800 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:33.435021 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 20:58:33.435107 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 20:58:33.448213 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:33.442837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41148 (local address 127.0.97.60:35711)
0929 20:58:33.443033 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.443039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.446726 (+ 3687us) server_negotiation.cc:407] Connection header received
0929 20:58:33.446915 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.446922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.447020 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.447140 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:33.447983 (+ 843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41148: BlockingRecv error: recv got EOF from 127.0.0.1:41148 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:58:33.462472 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:33.455665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60850 (local address 127.0.97.61:34009)
0929 20:58:33.455865 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.455872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.456427 (+ 555us) server_negotiation.cc:407] Connection header received
0929 20:58:33.456614 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.456620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.456760 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.456858 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:58:33.457294 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.458515 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.460446 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.461078 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.461207 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:33.461676 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:58:33.461805 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:58:33.461897 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:33.462069 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 20:58:33.462153 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 20:58:33.475275 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:33.472866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41154 (local address 127.0.97.60:35711)
0929 20:58:33.473139 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.473146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.473694 (+ 548us) server_negotiation.cc:407] Connection header received
0929 20:58:33.473861 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.473867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.473972 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.474088 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:33.475025 (+ 937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41154: BlockingRecv error: recv got EOF from 127.0.0.1:41154 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
I20260929 20:58:33.493734 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:33.486735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60854 (local address 127.0.97.61:34009)
0929 20:58:33.486962 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.486968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.487466 (+ 498us) server_negotiation.cc:407] Connection header received
0929 20:58:33.487668 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.487678 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.487768 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.487906 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:58:33.488324 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.489543 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.491563 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.492334 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.492471 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:33.492910 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:58:33.493065 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:58:33.493158 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:33.493301 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 20:58:33.493385 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260929 20:58:33.505782 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:33.503663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.0.97.60:35711)
0929 20:58:33.503866 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.503873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.504375 (+ 502us) server_negotiation.cc:407] Connection header received
0929 20:58:33.504548 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.504554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.504656 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.504767 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:33.505569 (+ 802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41160: BlockingRecv error: recv got EOF from 127.0.0.1:41160 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:58:33.532680 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:33.525766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60864 (local address 127.0.97.61:34009)
0929 20:58:33.525960 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.525966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.526531 (+ 565us) server_negotiation.cc:407] Connection header received
0929 20:58:33.526716 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.526722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.526811 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.526915 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:58:33.527364 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.528599 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.530575 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.531244 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.531375 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:33.531889 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:58:33.532029 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:58:33.532112 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:33.532283 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 20:58:33.532378 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 20:58:33.541630 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:33.540057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41170 (local address 127.0.97.60:35711)
0929 20:58:33.540227 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.540233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.540253 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:58:33.540333 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.540338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.540427 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.540532 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:33.541416 (+ 884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41170: BlockingRecv error: recv got EOF from 127.0.0.1:41170 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:58:33.582882 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:33.576958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60872 (local address 127.0.97.61:34009)
0929 20:58:33.577162 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.577173 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.577514 (+ 341us) server_negotiation.cc:407] Connection header received
0929 20:58:33.577700 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.577706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.577831 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.577982 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:58:33.578469 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.579584 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.580966 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.581633 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.581765 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:33.582080 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:58:33.582255 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:58:33.582349 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:33.582451 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 20:58:33.582538 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 20:58:33.590675 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:33.588707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.0.97.60:35711)
0929 20:58:33.588909 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.588915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.589235 (+ 320us) server_negotiation.cc:407] Connection header received
0929 20:58:33.589428 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.589432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.589513 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.589581 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:33.590441 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41172: BlockingRecv error: recv got EOF from 127.0.0.1:41172 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:58:33.666879 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:33.662284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60880 (local address 127.0.97.61:34009)
0929 20:58:33.662555 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.662562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.662733 (+ 171us) server_negotiation.cc:407] Connection header received
0929 20:58:33.662910 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.662914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.662981 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.663064 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:58:33.663387 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.664131 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.665419 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.665833 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.665936 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:33.666279 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:58:33.666387 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:58:33.666450 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:33.666551 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 20:58:33.666615 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260929 20:58:33.674093 22947 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:33.672414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.0.97.60:35711)
0929 20:58:33.672589 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.672595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.672859 (+ 264us) server_negotiation.cc:407] Connection header received
0929 20:58:33.673015 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.673019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.673090 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.673173 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:33.673857 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41174: BlockingRecv error: recv got EOF from 127.0.0.1:41174 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:58:33.785113 22961 catalog_manager.cc:1259] Loaded cluster ID: c93f51ab6c3b406eaf1c573994c30981
I20260929 20:58:33.785209 22961 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92: loading cluster ID for follower catalog manager: success
I20260929 20:58:33.787671 22961 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92: acquiring CA information for follower catalog manager: success
I20260929 20:58:33.789418 22961 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d620b93aea4e49e7b8d78042bcc15b92: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:58:33.810701 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:33.805596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60894 (local address 127.0.97.61:34009)
0929 20:58:33.805833 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.805858 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.806140 (+ 282us) server_negotiation.cc:407] Connection header received
0929 20:58:33.806386 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.806392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.806467 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.806549 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:58:33.806967 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.807794 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.809134 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.809535 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.809644 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:33.810080 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:58:33.810217 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:58:33.810291 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:33.810418 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 20:58:33.810484 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 20:58:33.821308 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:33.815909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41186 (local address 127.0.97.60:35711)
0929 20:58:33.816081 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.816085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.816325 (+ 240us) server_negotiation.cc:407] Connection header received
0929 20:58:33.816470 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.816474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.816633 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.816728 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:58:33.817058 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.817787 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.819106 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.819511 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.819602 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:33.820037 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:58:33.820731 (+ 694us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:58:33.820848 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:33.821022 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 20:58:33.821088 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:58:33.834064 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:33.826873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60906 (local address 127.0.97.61:34009)
0929 20:58:33.827268 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.827274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.827770 (+ 496us) server_negotiation.cc:407] Connection header received
0929 20:58:33.828064 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.828071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.828161 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.828221 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:58:33.829013 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.830174 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.832176 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.832774 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.832876 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:33.833397 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:58:33.833537 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:58:33.833616 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:33.833734 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 20:58:33.833815 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254}
I20260929 20:58:33.837977 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:33.826755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39164 (local address 127.0.97.62:43927)
0929 20:58:33.832042 (+ 5287us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.832048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.832066 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:58:33.832181 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.832187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.832397 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.832569 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:58:33.833131 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.834056 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.836003 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.836391 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.836534 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:33.836941 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:58:33.837386 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:58:33.837459 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:33.837592 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 20:58:33.837676 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5159,"thread_start_us":5075,"threads_started":1}
I20260929 20:58:33.838373 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:33.827048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.0.97.60:35711)
0929 20:58:33.827260 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:33.827266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:33.832459 (+ 5193us) server_negotiation.cc:407] Connection header received
0929 20:58:33.832644 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:33.832649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:33.832730 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:33.832790 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:58:33.833579 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.834739 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.836570 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:33.837072 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:33.837183 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:33.837581 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:58:33.837713 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:58:33.837795 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:33.837907 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 20:58:33.837991 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:58:33.843215 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 22729
I20260929 20:58:33.844282 22791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:58:33.953622 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 22729
I20260929 20:58:33.971223 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 22803
I20260929 20:58:33.972314 22865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:58:34.078650 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 22803
I20260929 20:58:34.094938 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 22877
I20260929 20:58:34.096024 22939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:58:34.200851 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 22877
2026-09-29T20:58:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4178 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:58:34 dist-test-slave-m9g1 krb5kdc[23125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:58:34 dist-test-slave-m9g1 krb5kdc[23125](info): set up 2 sockets
Sep 29 20:58:34 dist-test-slave-m9g1 krb5kdc[23125](info): commencing operation
krb5kdc: starting...
W20260929 20:58:35.210461 388 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/test-user.keytab.
Sep 29 20:58:35 dist-test-slave-m9g1 krb5kdc[23125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:58:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:58:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:58:35.461341 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46845
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46845,127.0.97.61:39507,127.0.97.60:39431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:58:35.782955 23138 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:58:35.783298 23138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:35.783397 23138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:35.783514 23138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:35.790715 23138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:35.790807 23138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:35.790871 23138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:35.790940 23138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:35.791023 23138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:35.800779 23138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46845,127.0.97.61:39507,127.0.97.60:39431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:35.802112 23138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:35.803417 23138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:58:35.814704 23143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:35.814738 23144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:35.814769 23146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:35.815568 23138 server_base.cc:1034] running on GCE node
I20260929 20:58:35.816376 23138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:35.817725 23138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:35.818934 23138 hybrid_clock.cc:648] HybridClock initialized: now 1790715515818858 us; error 50 us; skew 500 ppm
I20260929 20:58:35.819240 23138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:35 dist-test-slave-m9g1 krb5kdc[23125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:35.826290 23138 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:58:35.827644 23138 webserver.cc:460] Webserver started at http://127.0.97.62:33287/ using document root <none> and password file <none>
I20260929 20:58:35.828310 23138 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:35.828403 23138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:35.828711 23138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:35.831642 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-0/data/instance:
uuid: "4809e381a6e44d7b8a91c67877cf093a"
format_stamp: "Formatted at 2026-09-29 20:58:35 on dist-test-slave-m9g1"
server_key: "badde18a5093ece94f7de196d42c6c1d"
server_key_iv: "de4a1abdbc545bf46a06d3f688706ca8"
server_key_version: "encryptionkey@0"
I20260929 20:58:35.832388 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-0/wal/instance:
uuid: "4809e381a6e44d7b8a91c67877cf093a"
format_stamp: "Formatted at 2026-09-29 20:58:35 on dist-test-slave-m9g1"
server_key: "badde18a5093ece94f7de196d42c6c1d"
server_key_iv: "de4a1abdbc545bf46a06d3f688706ca8"
server_key_version: "encryptionkey@0"
I20260929 20:58:35.837342 23138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:58:35.840620 23153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:35.841897 23138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:58:35.842069 23138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-0/wal
uuid: "4809e381a6e44d7b8a91c67877cf093a"
format_stamp: "Formatted at 2026-09-29 20:58:35 on dist-test-slave-m9g1"
server_key: "badde18a5093ece94f7de196d42c6c1d"
server_key_iv: "de4a1abdbc545bf46a06d3f688706ca8"
server_key_version: "encryptionkey@0"
I20260929 20:58:35.842301 23138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:35.861001 23138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:35.865047 23138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:35.865305 23138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:35.882953 23138 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46845
I20260929 20:58:35.882956 23204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46845 every 8 connection(s)
I20260929 20:58:35.884349 23138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:58:35.884883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 23138
I20260929 20:58:35.885042 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-0/wal/instance
I20260929 20:58:35.885617 388 external_mini_cluster.cc:1428] Setting key 90f7cba07ab9c6c36557cbbcfe064637
I20260929 20:58:35.888209 23205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:35.894851 23205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 29 20:58:35 dist-test-slave-m9g1 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:58:35.917481 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:35.897798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46845 (local address 127.0.0.1:58216)
0929 20:58:35.898647 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:35.898696 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:58:35.900786 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:58:35.901788 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:58:35.901794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:58:35.902422 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:58:35.902769 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:35.902787 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:35.905051 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:35.905056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:35.906166 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:35.906178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:35.907152 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:35.907159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:35.907355 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:35.907927 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:58:35.907950 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:58:35.910389 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:58:35.913420 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:35.913432 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:35.913446 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:35.913813 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:35.914219 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:35.914224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:35.914228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:35.914384 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:35.915133 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:58:35.915141 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:58:35.915337 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 20:58:35.915559 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 20:58:35.916167 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":127,"thread_start_us":357,"threads_started":1}
I20260929 20:58:35.917520 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:35.899144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58216 (local address 127.0.97.62:46845)
0929 20:58:35.899789 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:35.899797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:35.899855 (+ 58us) server_negotiation.cc:407] Connection header received
0929 20:58:35.901193 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:35.901201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:35.901552 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:35.901773 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:35.902987 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:35.904861 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:35.906328 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:35.906922 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:35.907106 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:35.910632 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:35.910641 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:35.910653 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:35.910686 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:35.913290 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:35.913944 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:35.913951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:35.913953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:35.914095 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:35.914495 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:35.914498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:35.914500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:35.915003 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:35.915191 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:35.915714 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 20:58:35.916010 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":294,"threads_started":1}
I20260929 20:58:35.920810 23205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:58:35.921979 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:35.921580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39507 (local address 127.0.0.1:55140)
0929 20:58:35.921737 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:35.921840 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 20:58:35.922284 23157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39507: connect: Connection refused (error 111)
I20260929 20:58:35.923739 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:35.923400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39507 (local address 127.0.0.1:55156)
0929 20:58:35.923543 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:35.923616 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 20:58:35.923918 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39507: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39507: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 20:58:35.938607 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39507
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46845,127.0.97.61:39507,127.0.97.60:39431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:58:35.973270 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } attempt: 1
I20260929 20:58:35.974673 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:35.973988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39507 (local address 127.0.0.1:55160)
0929 20:58:35.974376 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:35.974507 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
I20260929 20:58:35.976231 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:35.975511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39507 (local address 127.0.0.1:55172)
0929 20:58:35.975961 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:35.976062 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326}
W20260929 20:58:35.976554 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39507: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39507: connect: Connection refused (error 111)
I20260929 20:58:36.044907 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } attempt: 2
I20260929 20:58:36.046064 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.045587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39507 (local address 127.0.0.1:55188)
0929 20:58:36.045829 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.045926 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 20:58:36.047158 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.046780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39507 (local address 127.0.0.1:55204)
0929 20:58:36.046942 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.047013 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 20:58:36.047343 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39507: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39507: connect: Connection refused (error 111)
I20260929 20:58:36.138571 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } attempt: 3
I20260929 20:58:36.139595 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.139179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39507 (local address 127.0.0.1:55218)
0929 20:58:36.139377 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.139461 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 20:58:36.140516 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.140179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39507 (local address 127.0.0.1:55226)
0929 20:58:36.140315 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.140380 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 20:58:36.140689 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39507: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39507: connect: Connection refused (error 111)
W20260929 20:58:36.261353 23212 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:58:36.261682 23212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:36.261775 23212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:36.261888 23212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:36.269032 23212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:36.269135 23212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:36.269215 23212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:36.269284 23212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:36.269367 23212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:36.279403 23212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46845,127.0.97.61:39507,127.0.97.60:39431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:36.280694 23212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:36.282054 23212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 20:58:36.283900 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } attempt: 4
I20260929 20:58:36.285003 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.284488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39507 (local address 127.0.0.1:55242)
0929 20:58:36.284778 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.284867 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260929 20:58:36.285962 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.285613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39507 (local address 127.0.0.1:55250)
0929 20:58:36.285756 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.285833 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 20:58:36.286162 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39507: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39507: connect: Connection refused (error 111)
W20260929 20:58:36.293560 23217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:36.293615 23220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:36.293627 23218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:36.294093 23212 server_base.cc:1034] running on GCE node
I20260929 20:58:36.294920 23212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:36.296326 23212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:36.297547 23212 hybrid_clock.cc:648] HybridClock initialized: now 1790715516297491 us; error 50 us; skew 500 ppm
I20260929 20:58:36.297864 23212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:36 dist-test-slave-m9g1 krb5kdc[23125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:36.300845 23212 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 20:58:36.302277 23212 webserver.cc:460] Webserver started at http://127.0.97.61:44209/ using document root <none> and password file <none>
I20260929 20:58:36.302958 23212 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:36.303054 23212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:36.303349 23212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:36.306211 23212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-1/data/instance:
uuid: "897bdbc6807c4ec5b9fcbb5691297897"
format_stamp: "Formatted at 2026-09-29 20:58:36 on dist-test-slave-m9g1"
server_key: "2d1a7f705f44decfdc7cf1f11c0120cb"
server_key_iv: "ad8af9c8fc00357c4d4a07a3c86e1579"
server_key_version: "encryptionkey@0"
I20260929 20:58:36.306981 23212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-1/wal/instance:
uuid: "897bdbc6807c4ec5b9fcbb5691297897"
format_stamp: "Formatted at 2026-09-29 20:58:36 on dist-test-slave-m9g1"
server_key: "2d1a7f705f44decfdc7cf1f11c0120cb"
server_key_iv: "ad8af9c8fc00357c4d4a07a3c86e1579"
server_key_version: "encryptionkey@0"
I20260929 20:58:36.311903 23212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:58:36.315292 23227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:36.316650 23212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:58:36.316808 23212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-1/wal
uuid: "897bdbc6807c4ec5b9fcbb5691297897"
format_stamp: "Formatted at 2026-09-29 20:58:36 on dist-test-slave-m9g1"
server_key: "2d1a7f705f44decfdc7cf1f11c0120cb"
server_key_iv: "ad8af9c8fc00357c4d4a07a3c86e1579"
server_key_version: "encryptionkey@0"
I20260929 20:58:36.317008 23212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:36.330057 23212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:36.334149 23212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:36.334447 23212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:36.351270 23212 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39507
I20260929 20:58:36.351264 23278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39507 every 8 connection(s)
I20260929 20:58:36.352612 23212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-1/data/info.pb
I20260929 20:58:36.356238 23279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:36.362736 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 23212
I20260929 20:58:36.362876 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-1/wal/instance
I20260929 20:58:36.362488 23279 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:58:36.363313 388 external_mini_cluster.cc:1428] Setting key 0730555a756ef4e5f656dbdb362b0ae1
Sep 29 20:58:36 dist-test-slave-m9g1 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:58:36.382485 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.365058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58224 (local address 127.0.97.62:46845)
0929 20:58:36.365218 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:36.365223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:36.366535 (+ 1312us) server_negotiation.cc:407] Connection header received
0929 20:58:36.368401 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:36.368407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:36.368487 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:36.368570 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:36.370050 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.370767 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.372453 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.372875 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.372986 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:36.376763 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:36.376768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:36.376771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:36.376807 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:36.379429 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:36.380187 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:36.380193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:36.380196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:36.380319 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:36.380724 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:36.380729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:36.380732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:36.381023 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:36.381157 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:36.381869 (+ 712us) server_negotiation.cc:299] Negotiation successful
0929 20:58:36.382107 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:58:36.383484 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.365617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46845 (local address 127.0.0.1:58224)
0929 20:58:36.366382 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.366420 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:58:36.368124 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:58:36.368815 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:58:36.368823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:58:36.369339 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:58:36.369880 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:36.369897 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.370924 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.370927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:36.372275 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:36.372298 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.373093 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.373100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:36.373368 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:36.374047 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:58:36.374079 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:58:36.376534 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:58:36.379582 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:36.379592 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:36.379612 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:36.380067 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:36.380431 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:36.380436 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:36.380440 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:36.380615 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:36.381147 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:58:36.381155 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:58:36.381519 (+ 364us) client_negotiation.cc:769] Sending connection context
0929 20:58:36.381773 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 20:58:36.382353 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":294,"threads_started":1}
I20260929 20:58:36.385234 23279 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 29 20:58:36 dist-test-slave-m9g1 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:58:36.399971 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.385980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39507 (local address 127.0.0.1:55256)
0929 20:58:36.386129 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.386143 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:58:36.386259 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:58:36.387367 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:58:36.387371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:58:36.387784 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:58:36.387917 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:36.387926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.389463 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.389467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:36.390327 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:36.390338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.391135 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.391141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:36.391315 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:36.391850 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:58:36.391866 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:58:36.393973 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:58:36.397582 (+ 3609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:36.397585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:36.397588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:36.397978 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:36.398367 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:36.398370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:36.398372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:36.398518 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:36.399263 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:58:36.399270 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:58:36.399381 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 20:58:36.399508 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 20:58:36.399709 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 20:58:36.400308 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.386221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55256 (local address 127.0.97.61:39507)
0929 20:58:36.386925 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:36.386931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:36.386957 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:58:36.386999 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:36.387005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:36.387192 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:36.387337 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:36.388077 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.389320 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.390446 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.390983 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.391091 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:36.394219 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:36.394236 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:36.394253 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:36.394294 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:36.397444 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:36.398104 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:36.398113 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:36.398116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:36.398275 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:36.398685 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:36.398691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:36.398694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:36.399120 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:36.399318 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:36.399689 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 20:58:36.399933 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":396,"threads_started":1}
I20260929 20:58:36.402609 23279 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:58:36.403681 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.403259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39606)
0929 20:58:36.403438 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.403551 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 20:58:36.403941 23229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
I20260929 20:58:36.405375 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.405079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39610)
0929 20:58:36.405207 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.405279 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 20:58:36.405568 23279 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39431: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 20:58:36.417438 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
/tmp/dist-test-taskPL7Duu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39431
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46845,127.0.97.61:39507,127.0.97.60:39431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:58:36.454852 23279 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } attempt: 1
I20260929 20:58:36.456037 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.455528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39624)
0929 20:58:36.455738 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.455849 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 20:58:36.456984 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.456627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39640)
0929 20:58:36.456758 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.456859 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 20:58:36.457168 23279 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39431: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
I20260929 20:58:36.525427 23279 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } attempt: 2
I20260929 20:58:36.526655 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.526102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39648)
0929 20:58:36.526342 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.526467 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 20:58:36.527608 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.527252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39662)
0929 20:58:36.527391 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.527473 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 20:58:36.527794 23279 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39431: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
I20260929 20:58:36.585420 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } attempt: 5
Sep 29 20:58:36 dist-test-slave-m9g1 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:58:36.599350 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.586234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.0.97.61:39507)
0929 20:58:36.586424 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:36.586430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:36.586453 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:58:36.586633 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:36.586636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:36.586716 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:36.586799 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:36.587743 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.588414 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.590173 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.590654 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.590751 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:36.594367 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:36.594372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:36.594375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:36.594408 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:36.596843 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:36.597651 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:36.597654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:36.597657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:36.597771 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:36.598134 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:36.598136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:36.598139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:36.598441 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:36.598551 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:36.598972 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 20:58:36.599134 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:58:36.599525 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.586081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39507 (local address 127.0.0.1:55270)
0929 20:58:36.586346 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.586370 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:58:36.586512 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:58:36.586874 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:58:36.586879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:58:36.587379 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:58:36.587610 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:36.587623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.588578 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.588584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:36.590007 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:36.590026 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.590825 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.590831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:36.591044 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:36.591674 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:58:36.591697 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:58:36.594156 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:58:36.596985 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:36.596991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:36.596995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:36.597535 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:36.597873 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:36.597878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:36.597881 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:36.598046 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:36.598554 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:58:36.598560 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:58:36.598729 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 20:58:36.598901 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 20:58:36.599186 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 20:58:36.600636 23205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:58:36.601779 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.601283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39676)
0929 20:58:36.601481 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.601585 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 20:58:36.602728 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.602376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39682)
0929 20:58:36.602518 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.602611 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 20:58:36.602962 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39431: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
I20260929 20:58:36.618994 23279 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } attempt: 3
I20260929 20:58:36.620092 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.619626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39686)
0929 20:58:36.619828 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.619917 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 20:58:36.621111 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.620679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39698)
0929 20:58:36.620888 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.620992 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260929 20:58:36.621271 23279 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39431: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
I20260929 20:58:36.654219 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } attempt: 1
I20260929 20:58:36.655427 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.654875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39700)
0929 20:58:36.655119 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.655226 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 20:58:36.656515 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.656152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39712)
0929 20:58:36.656293 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.656382 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 20:58:36.656719 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39431: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
I20260929 20:58:36.724960 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } attempt: 2
I20260929 20:58:36.726128 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.725585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39716)
0929 20:58:36.725809 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.725927 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 20:58:36.727108 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.726744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39722)
0929 20:58:36.726889 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.726972 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 20:58:36.727303 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39431: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
W20260929 20:58:36.741652 23286 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:58:36.741942 23286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:58:36.742009 23286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:58:36.742085 23286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:58:36.749379 23286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:58:36.749462 23286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:58:36.749516 23286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:58:36.749563 23286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:58:36.749622 23286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:58:36.759409 23286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46845,127.0.97.61:39507,127.0.97.60:39431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:58:36.760584 23286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:58:36.761796 23286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 20:58:36.764460 23279 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } attempt: 4
I20260929 20:58:36.765424 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.764995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39738)
0929 20:58:36.765200 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.765283 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 20:58:36.766513 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.766113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39746)
0929 20:58:36.766297 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.766387 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 20:58:36.766772 23279 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39431: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
W20260929 20:58:36.772784 23294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:36.772959 23291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:58:36.773082 23292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:58:36.773524 23286 server_base.cc:1034] running on GCE node
I20260929 20:58:36.774223 23286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:58:36.775557 23286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:58:36.776787 23286 hybrid_clock.cc:648] HybridClock initialized: now 1790715516776719 us; error 41 us; skew 500 ppm
I20260929 20:58:36.777114 23286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:58:36 dist-test-slave-m9g1 krb5kdc[23125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:58:36.779992 23286 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 20:58:36.781373 23286 webserver.cc:460] Webserver started at http://127.0.97.60:42515/ using document root <none> and password file <none>
I20260929 20:58:36.782009 23286 fs_manager.cc:359] Metadata directory not provided
I20260929 20:58:36.782100 23286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:58:36.782436 23286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:58:36.785200 23286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-2/data/instance:
uuid: "934cec02822b4e67887359bd48fa9d7d"
format_stamp: "Formatted at 2026-09-29 20:58:36 on dist-test-slave-m9g1"
server_key: "a64de8ad76a08db864d21f61379d84ee"
server_key_iv: "8c1bebfb9dd52962d1fcca9ba4038b6e"
server_key_version: "encryptionkey@0"
I20260929 20:58:36.785923 23286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-2/wal/instance:
uuid: "934cec02822b4e67887359bd48fa9d7d"
format_stamp: "Formatted at 2026-09-29 20:58:36 on dist-test-slave-m9g1"
server_key: "a64de8ad76a08db864d21f61379d84ee"
server_key_iv: "8c1bebfb9dd52962d1fcca9ba4038b6e"
server_key_version: "encryptionkey@0"
I20260929 20:58:36.790745 23286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:58:36.793862 23301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:58:36.795111 23286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 20:58:36.795260 23286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-2/wal
uuid: "934cec02822b4e67887359bd48fa9d7d"
format_stamp: "Formatted at 2026-09-29 20:58:36 on dist-test-slave-m9g1"
server_key: "a64de8ad76a08db864d21f61379d84ee"
server_key_iv: "8c1bebfb9dd52962d1fcca9ba4038b6e"
server_key_version: "encryptionkey@0"
I20260929 20:58:36.795460 23286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:58:36.815088 23286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:58:36.819002 23286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:58:36.819229 23286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:58:36.833495 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } attempt: 3
I20260929 20:58:36.834623 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.834066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39754)
0929 20:58:36.834322 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.834431 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 20:58:36.835726 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.835262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39764)
0929 20:58:36.835494 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.835591 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260929 20:58:36.835999 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39431: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39431: connect: Connection refused (error 111)
I20260929 20:58:36.836061 23286 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:39431
I20260929 20:58:36.836088 23352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:39431 every 8 connection(s)
I20260929 20:58:36.837347 23286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-2/data/info.pb
I20260929 20:58:36.840524 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu as pid 23286
I20260929 20:58:36.840677 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPL7Duu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715169435170-388-0/minicluster-data/master-2/wal/instance
I20260929 20:58:36.840950 23353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:58:36.841100 388 external_mini_cluster.cc:1428] Setting key 8c67c2875c8aa7924ef8354b1db7aec4
I20260929 20:58:36.850418 23353 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:58:36 dist-test-slave-m9g1 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 20:58:36 dist-test-slave-m9g1 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:58:36.863621 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.842688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39780 (local address 127.0.97.60:39431)
0929 20:58:36.843451 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:36.843466 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:36.843513 (+ 47us) server_negotiation.cc:407] Connection header received
0929 20:58:36.845503 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:36.845531 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:36.846071 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:36.846517 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:36.847705 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.849619 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.850890 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.851880 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.852059 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:36.856139 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:36.856149 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:36.856166 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:36.856208 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:36.859696 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:36.860314 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:36.860322 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:36.860331 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:36.860512 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:36.860876 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:36.860880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:36.860883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:36.861529 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:36.861782 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:36.862214 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 20:58:36.862575 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":296,"threads_started":1}
I20260929 20:58:36.867640 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.853096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.0.97.62:46845)
0929 20:58:36.853253 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:36.853257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:36.853921 (+ 664us) server_negotiation.cc:407] Connection header received
0929 20:58:36.854103 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:36.854109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:36.854179 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:36.854285 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:36.855357 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.856029 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.857926 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.858457 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.858548 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:36.861739 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:36.861745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:36.861749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:36.861788 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:36.864726 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:36.865469 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:36.865472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:36.865475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:36.865596 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:36.865970 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:36.865975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:36.865977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:36.866299 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:36.866433 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:36.867033 (+ 600us) server_negotiation.cc:299] Negotiation successful
0929 20:58:36.867245 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 20:58:36.868338 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.853129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46845 (local address 127.0.0.1:58230)
0929 20:58:36.853793 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.853827 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 20:58:36.853986 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:58:36.854338 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:58:36.854346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:58:36.854926 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:58:36.855164 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:36.855172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.856166 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.856171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:36.857770 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:36.857785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.858643 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.858650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:36.858812 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:36.859338 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:58:36.859360 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:58:36.861569 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:58:36.864840 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:36.864849 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:36.864855 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:36.865361 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:36.865697 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:36.865702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:36.865705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:36.865870 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:36.866479 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:58:36.866486 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:58:36.866672 (+ 186us) client_negotiation.cc:769] Sending connection context
0929 20:58:36.867808 (+ 1136us) client_negotiation.cc:240] Negotiation successful
0929 20:58:36.868084 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":363,"threads_started":1}
I20260929 20:58:36.869655 23353 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:58:36 dist-test-slave-m9g1 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 29 20:58:36 dist-test-slave-m9g1 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:58:36.881553 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.866634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.0.97.62:46845)
0929 20:58:36.867163 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:36.867168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:36.867188 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:58:36.867231 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:36.867235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:36.867297 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:36.867397 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:36.868474 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.869464 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.871084 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.871500 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.871607 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:36.875767 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:36.875772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:36.875775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:36.875808 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:36.878893 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:36.879662 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:36.879668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:36.879672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:36.879825 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:36.880135 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:36.880140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:36.880144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:36.880469 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:36.880645 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:36.880981 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 20:58:36.881200 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":238,"threads_started":1}
I20260929 20:58:36.882844 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.870488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55274 (local address 127.0.97.61:39507)
0929 20:58:36.870667 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:36.870673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:36.870796 (+ 123us) server_negotiation.cc:407] Connection header received
0929 20:58:36.870949 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:36.870953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:36.871014 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:36.871096 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:36.871746 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.872571 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.873789 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.874239 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.874346 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:36.877412 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:36.877419 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:36.877423 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:36.877463 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:36.880271 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:36.880935 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:36.880940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:36.880944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:36.881101 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:36.881489 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:36.881492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:36.881494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:36.881775 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:36.881907 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:36.882362 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 20:58:36.882575 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 20:58:36.882818 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.870562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39507 (local address 127.0.0.1:55274)
0929 20:58:36.870717 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.870734 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:58:36.870846 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:58:36.871132 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:58:36.871136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:58:36.871478 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:58:36.871605 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:36.871613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.872724 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.872729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:36.873652 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:36.873663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.874369 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.874374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:36.874542 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:36.875005 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:58:36.875027 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:58:36.877208 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:58:36.880391 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:36.880395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:36.880397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:36.880829 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:36.881219 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:36.881223 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:36.881226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:36.881380 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:36.881904 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:58:36.881908 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:58:36.882069 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 20:58:36.882255 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 20:58:36.882517 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 20:58:36.883805 23353 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:58:36 dist-test-slave-m9g1 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 20:58:36.895483 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.884523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.0.97.60:39431)
0929 20:58:36.884680 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:36.884685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:36.884806 (+ 121us) server_negotiation.cc:407] Connection header received
0929 20:58:36.884941 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:36.884945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:36.884997 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:36.885066 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:36.885676 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.886367 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.887416 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.887822 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.887924 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:36.890764 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:36.890770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:36.890773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:36.890798 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:36.893305 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:36.893824 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:36.893829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:36.893832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:36.893924 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:36.894267 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:36.894270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:36.894272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:36.894516 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:36.894631 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:36.894940 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 20:58:36.895118 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:58:36.895669 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.884538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39790)
0929 20:58:36.884742 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:36.884755 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 20:58:36.884844 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:58:36.885094 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:58:36.885098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:58:36.885435 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:58:36.885560 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:36.885568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.886470 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.886475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:36.887299 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:36.887309 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.887969 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.887974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:36.888139 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:36.888624 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:58:36.888639 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:58:36.890616 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:58:36.893414 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:36.893419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:36.893423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:36.893730 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:36.894024 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:36.894028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:36.894032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:36.894139 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:36.894615 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:58:36.894619 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:58:36.894739 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 20:58:36.894854 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 20:58:36.895281 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":125}
Sep 29 20:58:36 dist-test-slave-m9g1 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:58:36.900465 23353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d: Bootstrap starting.
I20260929 20:58:36.903010 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:36.889762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55290 (local address 127.0.97.61:39507)
0929 20:58:36.889929 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:36.889934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:36.889952 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:58:36.890045 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:36.890050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:36.890117 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:36.890234 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:36.891712 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.892718 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.893812 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:36.894254 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:36.894367 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:36.898296 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:36.898301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:36.898304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:36.898340 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:36.900692 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:36.901254 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:36.901260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:36.901263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:36.901390 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:36.901722 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:36.901727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:36.901730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:36.901994 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:36.902256 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:36.902521 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 20:58:36.902723 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:58:36.903663 23353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:36.904850 23353 log.cc:824] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:36.907548 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:58:36.907812 23353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d: No bootstrap required, opened a new log
I20260929 20:58:36.909713 23180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4809e381a6e44d7b8a91c67877cf093a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58236
I20260929 20:58:36.914916 23353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4809e381a6e44d7b8a91c67877cf093a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } } peers { permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } } peers { permanent_uuid: "934cec02822b4e67887359bd48fa9d7d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } }
I20260929 20:58:36.915236 23353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 934cec02822b4e67887359bd48fa9d7d, State: Initialized, Role: FOLLOWER
I20260929 20:58:36.915657 23353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4809e381a6e44d7b8a91c67877cf093a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } } peers { permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } } peers { permanent_uuid: "934cec02822b4e67887359bd48fa9d7d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } }
I20260929 20:58:36.916456 23360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4809e381a6e44d7b8a91c67877cf093a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } } peers { permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } } peers { permanent_uuid: "934cec02822b4e67887359bd48fa9d7d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } } }
I20260929 20:58:36.916743 23360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:58:36.917692 23353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d [sys.catalog]: configured and running, proceeding with master startup.
W20260929 20:58:36.926641 23371 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:58:36.926815 23371 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:58:36.924572 23286 master_runner.cc:186] Error getting master registration for 127.0.97.62:46845: OK, instance_id { permanent_uuid: "4809e381a6e44d7b8a91c67877cf093a" instance_seqno: 1790715515877933 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:58:36.935593 23286 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:58:37.013216 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } attempt: 4
Sep 29 20:58:37 dist-test-slave-m9g1 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 20:58:37.029237 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:37.013969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39798)
0929 20:58:37.014282 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:37.014302 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:58:37.014447 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:58:37.014811 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:58:37.014817 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:58:37.015350 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:58:37.015554 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:37.015567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:37.016971 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:37.016977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:37.018546 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:37.018563 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:37.019522 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:37.019529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:37.019748 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:37.020353 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:58:37.020377 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:58:37.023260 (+ 2883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:58:37.026551 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:37.026557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:37.026561 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:37.027055 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:37.027455 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:37.027461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:37.027465 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:37.027620 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:37.028198 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:58:37.028204 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:58:37.028406 (+ 202us) client_negotiation.cc:769] Sending connection context
0929 20:58:37.028552 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 20:58:37.028849 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260929 20:58:37.029238 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:37.014095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.0.97.60:39431)
0929 20:58:37.014297 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:37.014301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:37.014387 (+ 86us) server_negotiation.cc:407] Connection header received
0929 20:58:37.014566 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:37.014572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:37.014655 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:37.014764 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:37.015698 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:37.016814 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:37.018692 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:37.019366 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:37.019493 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:37.023487 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:37.023493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:37.023496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:37.023537 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:37.026422 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:37.027175 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:37.027181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:37.027185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:37.027349 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:37.027719 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:37.027724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:37.027727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:37.028072 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:37.028236 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:37.028661 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 20:58:37.028875 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 20:58:37.035166 23205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a: Bootstrap starting.
I20260929 20:58:37.039296 23205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:37.040666 23205 log.cc:824] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:37.044081 23205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a: No bootstrap required, opened a new log
I20260929 20:58:37.051823 23205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4809e381a6e44d7b8a91c67877cf093a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } } peers { permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } } peers { permanent_uuid: "934cec02822b4e67887359bd48fa9d7d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } }
I20260929 20:58:37.052268 23205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4809e381a6e44d7b8a91c67877cf093a, State: Initialized, Role: FOLLOWER
I20260929 20:58:37.052819 23205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4809e381a6e44d7b8a91c67877cf093a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } } peers { permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } } peers { permanent_uuid: "934cec02822b4e67887359bd48fa9d7d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } }
I20260929 20:58:37.053822 23374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4809e381a6e44d7b8a91c67877cf093a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } } peers { permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } } peers { permanent_uuid: "934cec02822b4e67887359bd48fa9d7d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } } }
I20260929 20:58:37.054239 23374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:58:37.055177 23205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:37.055229 23180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 20:58:37.055444 23180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:37.057842 23180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4809e381a6e44d7b8a91c67877cf093a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } } peers { permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } } peers { permanent_uuid: "934cec02822b4e67887359bd48fa9d7d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } }
I20260929 20:58:37.059113 23180 leader_election.cc:288] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [CANDIDATE]: Term 1 election: Requested vote from peers 897bdbc6807c4ec5b9fcbb5691297897 (127.0.97.61:39507), 934cec02822b4e67887359bd48fa9d7d (127.0.97.60:39431)
I20260929 20:58:37.060679 23328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4809e381a6e44d7b8a91c67877cf093a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "934cec02822b4e67887359bd48fa9d7d"
I20260929 20:58:37.061157 23328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:37.064814 23328 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4809e381a6e44d7b8a91c67877cf093a in term 1.
I20260929 20:58:37.065573 23155 leader_election.cc:302] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4809e381a6e44d7b8a91c67877cf093a, 934cec02822b4e67887359bd48fa9d7d; no voters:
I20260929 20:58:37.065953 23279 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } attempt: 5
I20260929 20:58:37.066319 23374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [term 1 FOLLOWER]: Leader election won for term 1
W20260929 20:58:37.072826 23307 tablet.cc:2369] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 20:58:37 dist-test-slave-m9g1 krb5kdc[23125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 20:58:37.085567 23374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [term 1 LEADER]: Becoming Leader. State: Replica: 4809e381a6e44d7b8a91c67877cf093a, State: Running, Role: LEADER
I20260929 20:58:37.086258 23374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4809e381a6e44d7b8a91c67877cf093a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } } peers { permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } } peers { permanent_uuid: "934cec02822b4e67887359bd48fa9d7d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } }
I20260929 20:58:37.090114 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:37.067046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39431 (local address 127.0.0.1:39800)
0929 20:58:37.067479 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 20:58:37.067499 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:58:37.067628 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:58:37.068473 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:58:37.068479 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:58:37.068994 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:58:37.069212 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:37.069224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:37.070867 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:37.070873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:37.072388 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:58:37.072406 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:58:37.074962 (+ 2556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:58:37.074970 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:58:37.075192 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:37.075963 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:58:37.075988 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:58:37.079282 (+ 3294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:58:37.084796 (+ 5514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:37.084802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:37.084826 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:37.085568 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:37.086415 (+ 847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:58:37.086420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:58:37.086445 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:58:37.086855 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:58:37.087869 (+ 1014us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:58:37.087873 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:58:37.088102 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 20:58:37.088317 (+ 215us) client_negotiation.cc:240] Negotiation successful
0929 20:58:37.088711 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297}
I20260929 20:58:37.091503 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:37.067843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39800 (local address 127.0.97.60:39431)
0929 20:58:37.068011 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:37.068017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:37.068035 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:58:37.068092 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:37.068101 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:37.068174 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:37.068266 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:37.069407 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:37.070675 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:37.073801 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:37.074572 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:37.074693 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:37.079561 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:37.079569 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:37.079575 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:37.079616 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:37.084584 (+ 4968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:37.085921 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:37.085927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:37.085930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:37.086223 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:37.087063 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:37.087068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:37.087071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:37.087666 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:37.088900 (+ 1234us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:37.089399 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 20:58:37.090177 (+ 778us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:58:37.097563 23279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897: Bootstrap starting.
I20260929 20:58:37.098906 23380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4809e381a6e44d7b8a91c67877cf093a. Latest consensus state: current_term: 1 leader_uuid: "4809e381a6e44d7b8a91c67877cf093a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4809e381a6e44d7b8a91c67877cf093a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } } peers { permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } } peers { permanent_uuid: "934cec02822b4e67887359bd48fa9d7d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } } }
I20260929 20:58:37.099346 23380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:37.105314 23279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897: Neither blocks nor log segments found. Creating new log.
I20260929 20:58:37.107640 23279 log.cc:824] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897: Log is configured to *not* fsync() on all Append() calls
I20260929 20:58:37.109620 23387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:58:37.112991 23390 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 20:58:37.118455 23279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897: No bootstrap required, opened a new log
W20260929 20:58:37.120589 23159 tablet.cc:2369] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:58:37.117437 23138 master_runner.cc:186] Error getting master registration for 127.0.97.61:39507: OK, instance_id { permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" instance_seqno: 1790715516346439 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:58:37.128666 23138 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:58:37.129514 23279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4809e381a6e44d7b8a91c67877cf093a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } } peers { permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } } peers { permanent_uuid: "934cec02822b4e67887359bd48fa9d7d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } }
I20260929 20:58:37.129902 23279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897bdbc6807c4ec5b9fcbb5691297897, State: Initialized, Role: FOLLOWER
I20260929 20:58:37.130481 23279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4809e381a6e44d7b8a91c67877cf093a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } } peers { permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } } peers { permanent_uuid: "934cec02822b4e67887359bd48fa9d7d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } }
I20260929 20:58:37.132858 23387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:58:37.157277 23395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4809e381a6e44d7b8a91c67877cf093a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } } peers { permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } } peers { permanent_uuid: "934cec02822b4e67887359bd48fa9d7d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } } }
I20260929 20:58:37.157692 23395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:58:37.160480 23328 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d [term 1 FOLLOWER]: Refusing update from remote peer 4809e381a6e44d7b8a91c67877cf093a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:58:37.162098 23374 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [LEADER]: Connected to new peer: Peer: permanent_uuid: "934cec02822b4e67887359bd48fa9d7d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:58:37.162317 23279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:58:37.162688 23253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:58:37.166169 23253 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897 [term 1 FOLLOWER]: Refusing update from remote peer 4809e381a6e44d7b8a91c67877cf093a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:58:37.166406 23254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4809e381a6e44d7b8a91c67877cf093a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "897bdbc6807c4ec5b9fcbb5691297897"
I20260929 20:58:37.167356 23374 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [LEADER]: Connected to new peer: Peer: permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:58:37.169960 23254 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4809e381a6e44d7b8a91c67877cf093a in term 1.
I20260929 20:58:37.172991 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:37.134312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55294 (local address 127.0.97.61:39507)
0929 20:58:37.134508 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:37.134514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:37.140266 (+ 5752us) server_negotiation.cc:407] Connection header received
0929 20:58:37.141239 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:37.141246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:37.141348 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:37.141422 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:37.148423 (+ 7001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:37.149429 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:37.152248 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:37.152899 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:37.153022 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:37.158268 (+ 5246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:37.158274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:37.158278 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:37.158319 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:37.161929 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:37.169287 (+ 7358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:37.169292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:37.169296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:37.169458 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:37.170978 (+ 1520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:37.170983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:37.170986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:37.171394 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:37.171624 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:37.172213 (+ 589us) server_negotiation.cc:299] Negotiation successful
0929 20:58:37.172493 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:58:37.173197 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:37.134462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.0.97.62:46845)
0929 20:58:37.134649 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:37.134654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:37.140767 (+ 6113us) server_negotiation.cc:407] Connection header received
0929 20:58:37.144240 (+ 3473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:37.144247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:37.144342 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:37.144414 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:37.146655 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:37.147699 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:37.150433 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:37.151052 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:37.151173 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:37.163657 (+ 12484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:37.163663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:37.163668 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:37.163709 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:37.166931 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:37.168696 (+ 1765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:37.168701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:37.168705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:37.168850 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:37.169881 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:37.169887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:37.169891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:37.170300 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:37.170544 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:37.172637 (+ 2093us) server_negotiation.cc:299] Negotiation successful
0929 20:58:37.172887 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 20:58:37.192164 23395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4809e381a6e44d7b8a91c67877cf093a. Latest consensus state: current_term: 1 leader_uuid: "4809e381a6e44d7b8a91c67877cf093a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4809e381a6e44d7b8a91c67877cf093a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } } peers { permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } } peers { permanent_uuid: "934cec02822b4e67887359bd48fa9d7d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } } }
I20260929 20:58:37.192468 23395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:58:37.206377 23402 mvcc.cc:205] Tried to move back new op lower bound from 7334770758241079296 to 7334770757991743488. Current Snapshot: MvccSnapshot[applied={T|T < 7334770758241079296 or (T in {7334770758241079296})}]
I20260929 20:58:37.207446 23360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4809e381a6e44d7b8a91c67877cf093a. Latest consensus state: current_term: 1 leader_uuid: "4809e381a6e44d7b8a91c67877cf093a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4809e381a6e44d7b8a91c67877cf093a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } } peers { permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } } peers { permanent_uuid: "934cec02822b4e67887359bd48fa9d7d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } } }
I20260929 20:58:37.207765 23360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:58:37.208199 23395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4809e381a6e44d7b8a91c67877cf093a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4809e381a6e44d7b8a91c67877cf093a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } } peers { permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } } peers { permanent_uuid: "934cec02822b4e67887359bd48fa9d7d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } } }
I20260929 20:58:37.208873 23395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:58:37.210283 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:37.134958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39804 (local address 127.0.97.60:39431)
0929 20:58:37.135137 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:37.135143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:37.153291 (+ 18148us) server_negotiation.cc:407] Connection header received
0929 20:58:37.153438 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:37.153443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:37.153527 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:37.153616 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:37.156761 (+ 3145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:37.157820 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:37.187100 (+ 29280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:37.187789 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:37.187891 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:37.201215 (+ 13324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:58:37.201223 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:58:37.201228 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:58:37.201286 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:58:37.205104 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:37.206796 (+ 1692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:37.206803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:37.206807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:37.206977 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:58:37.207870 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:58:37.207876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:58:37.207879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:58:37.208296 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:58:37.208537 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:37.209565 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0929 20:58:37.209865 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 20:58:37.211869 23401 mvcc.cc:205] Tried to move back new op lower bound from 7334770758241079296 to 7334770757991743488. Current Snapshot: MvccSnapshot[applied={T|T < 7334770758241079296}]
I20260929 20:58:37.212466 23360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4809e381a6e44d7b8a91c67877cf093a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4809e381a6e44d7b8a91c67877cf093a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } } peers { permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } } peers { permanent_uuid: "934cec02822b4e67887359bd48fa9d7d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } } }
I20260929 20:58:37.212713 23360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:58:37.217685 23387 catalog_manager.cc:1347] Generated new cluster ID: ac4b4d541a1a4039a2983ed0950c908b
I20260929 20:58:37.217809 23387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:58:37.217594 23380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4809e381a6e44d7b8a91c67877cf093a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4809e381a6e44d7b8a91c67877cf093a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } } peers { permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } } peers { permanent_uuid: "934cec02822b4e67887359bd48fa9d7d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } } }
I20260929 20:58:37.217886 23380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:37.218151 23380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4809e381a6e44d7b8a91c67877cf093a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4809e381a6e44d7b8a91c67877cf093a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46845 } } peers { permanent_uuid: "897bdbc6807c4ec5b9fcbb5691297897" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39507 } } peers { permanent_uuid: "934cec02822b4e67887359bd48fa9d7d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39431 } } }
I20260929 20:58:37.218402 23380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a [sys.catalog]: This master's current role is: LEADER
I20260929 20:58:37.245398 23417 catalog_manager.cc:1259] Loaded cluster ID: ac4b4d541a1a4039a2983ed0950c908b
I20260929 20:58:37.245507 23417 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897: loading cluster ID for follower catalog manager: success
W20260929 20:58:37.246622 23417 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260929 20:58:37.246743 23417 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:58:37.280745 23387 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:58:37.282810 23387 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:58:37.317601 23387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4809e381a6e44d7b8a91c67877cf093a: Generated new TSK 0
I20260929 20:58:37.323864 23387 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 20:58:37.456692 23356 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:37.440962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39820 (local address 127.0.97.60:39431)
0929 20:58:37.441286 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:37.441292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:37.454796 (+ 13504us) server_negotiation.cc:407] Connection header received
0929 20:58:37.455098 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:37.455104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:37.455194 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:37.455267 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:37.456431 (+ 1164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39820: BlockingRecv error: recv got EOF from 127.0.0.1:39820 (error 108)
Metrics: {"server-negotiator.queue_time_us":217}
W20260929 20:58:37.456862 23283 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:37.446678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55300 (local address 127.0.97.61:39507)
0929 20:58:37.446855 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:37.446861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:37.455276 (+ 8415us) server_negotiation.cc:407] Connection header received
0929 20:58:37.455363 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:37.455369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:37.455477 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:37.455585 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:37.456630 (+ 1045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55300: BlockingRecv error: recv got EOF from 127.0.0.1:55300 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 20:58:37.458539 23358 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:37.440790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.0.97.62:46845)
0929 20:58:37.441153 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:37.441159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:37.454651 (+ 13492us) server_negotiation.cc:407] Connection header received
0929 20:58:37.454958 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:37.454965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:37.455056 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:37.455128 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:37.458282 (+ 3154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58250: BlockingRecv error: recv got EOF from 127.0.0.1:58250 (error 108)
Metrics: {"server-negotiator.queue_time_us":231}
W20260929 20:58:37.464593 23283 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:37.462259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55306 (local address 127.0.97.61:39507)
0929 20:58:37.462435 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:37.462441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:37.462901 (+ 460us) server_negotiation.cc:407] Connection header received
0929 20:58:37.463173 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:37.463179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:37.463336 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:37.463413 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:37.464317 (+ 904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55306: BlockingRecv error: recv got EOF from 127.0.0.1:55306 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 20:58:37.470116 23283 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:37.468310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55312 (local address 127.0.97.61:39507)
0929 20:58:37.468470 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:37.468475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:37.468913 (+ 438us) server_negotiation.cc:407] Connection header received
0929 20:58:37.469067 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:37.469072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:37.469154 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:37.469252 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:37.469916 (+ 664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55312: BlockingRecv error: recv got EOF from 127.0.0.1:55312 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 20:58:37.476992 23283 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:37.474989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.0.97.61:39507)
0929 20:58:37.475147 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:37.475152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:37.475642 (+ 490us) server_negotiation.cc:407] Connection header received
0929 20:58:37.475809 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:37.475814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:37.475921 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:37.476024 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:37.476746 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55324: BlockingRecv error: recv got EOF from 127.0.0.1:55324 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 20:58:37.485846 23283 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:37.483811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55338 (local address 127.0.97.61:39507)
0929 20:58:37.484020 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:37.484025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:37.484498 (+ 473us) server_negotiation.cc:407] Connection header received
0929 20:58:37.484669 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:37.484674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:37.484763 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:37.484863 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:37.485613 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55338: BlockingRecv error: recv got EOF from 127.0.0.1:55338 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260929 20:58:37.532477 23283 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:37.523401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55344 (local address 127.0.97.61:39507)
0929 20:58:37.523612 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:37.523618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:37.530536 (+ 6918us) server_negotiation.cc:407] Connection header received
0929 20:58:37.531044 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:37.531051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:37.531152 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:37.531226 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:37.532218 (+ 992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55344: BlockingRecv error: recv got EOF from 127.0.0.1:55344 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 20:58:37.598318 23283 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:37.595789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.0.97.61:39507)
0929 20:58:37.595978 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:37.595985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:37.596527 (+ 542us) server_negotiation.cc:407] Connection header received
0929 20:58:37.596849 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:37.596855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:37.596948 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:37.597024 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:37.598040 (+ 1016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55348: BlockingRecv error: recv got EOF from 127.0.0.1:55348 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 20:58:37.680119 23283 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:37.670714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55362 (local address 127.0.97.61:39507)
0929 20:58:37.670946 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:37.670953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:37.677982 (+ 7029us) server_negotiation.cc:407] Connection header received
0929 20:58:37.678537 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:37.678547 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:37.678647 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:37.678727 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:37.679814 (+ 1087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55362: BlockingRecv error: recv got EOF from 127.0.0.1:55362 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260929 20:58:37.754531 23283 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:37.752185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55374 (local address 127.0.97.61:39507)
0929 20:58:37.752411 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:37.752418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:37.753020 (+ 602us) server_negotiation.cc:407] Connection header received
0929 20:58:37.753218 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:37.753224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:37.753335 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:37.753445 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:37.754306 (+ 861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55374: BlockingRecv error: recv got EOF from 127.0.0.1:55374 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 20:58:37.887404 23283 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:37.885525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55382 (local address 127.0.97.61:39507)
0929 20:58:37.885706 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:37.885712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:37.886040 (+ 328us) server_negotiation.cc:407] Connection header received
0929 20:58:37.886269 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:37.886273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:37.886349 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:37.886445 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:37.887175 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55382: BlockingRecv error: recv got EOF from 127.0.0.1:55382 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:58:37.928702 23371 catalog_manager.cc:1259] Loaded cluster ID: ac4b4d541a1a4039a2983ed0950c908b
I20260929 20:58:37.928812 23371 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d: loading cluster ID for follower catalog manager: success
I20260929 20:58:37.931411 23371 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d: acquiring CA information for follower catalog manager: success
I20260929 20:58:37.933145 23371 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 934cec02822b4e67887359bd48fa9d7d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 20:58:38.148211 23283 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:58:38.146507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55392 (local address 127.0.97.61:39507)
0929 20:58:38.146701 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:38.146707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:38.146919 (+ 212us) server_negotiation.cc:407] Connection header received
0929 20:58:38.147113 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:38.147118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:38.147193 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:38.147275 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:58:38.147998 (+ 723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55392: BlockingRecv error: recv got EOF from 127.0.0.1:55392 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:58:38.249974 23417 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897: acquiring CA information for follower catalog manager: success
I20260929 20:58:38.251865 23417 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 897bdbc6807c4ec5b9fcbb5691297897: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:58:38.669574 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:38.663920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55406 (local address 127.0.97.61:39507)
0929 20:58:38.664659 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:38.664664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:38.664681 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:58:38.664742 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:38.664745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:38.664920 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:38.665028 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:58:38.665501 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:38.666247 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:38.667557 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:38.667970 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:38.668062 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:38.668494 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:58:38.669010 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:58:38.669117 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:38.669213 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 20:58:38.669282 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":532,"threads_started":1}
I20260929 20:58:38.681516 23490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:38.675921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39836 (local address 127.0.97.60:39431)
0929 20:58:38.676338 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:38.676343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:38.676381 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:58:38.676567 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:38.676571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:38.676720 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:38.676867 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:58:38.677251 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:38.677949 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:38.679336 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:38.679751 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:38.679854 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:38.680372 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:58:38.680932 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:58:38.681085 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:38.681176 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 20:58:38.681247 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":212,"threads_started":1}
I20260929 20:58:38.705510 23490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:38.687779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39850 (local address 127.0.97.60:39431)
0929 20:58:38.687949 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:38.687955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:38.699458 (+ 11503us) server_negotiation.cc:407] Connection header received
0929 20:58:38.699708 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:38.699714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:38.699804 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:38.699909 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:58:38.700640 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:38.701782 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:38.703331 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:38.703935 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:38.704061 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:38.704513 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:58:38.704652 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:58:38.704744 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:38.705138 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 20:58:38.705216 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:58:38.707116 23496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:38.687031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58256 (local address 127.0.97.62:46845)
0929 20:58:38.698626 (+ 11595us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:38.698635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:38.699351 (+ 716us) server_negotiation.cc:407] Connection header received
0929 20:58:38.700444 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:38.700449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:38.700670 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:38.700820 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:58:38.701680 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:38.702795 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:38.704620 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:38.705196 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:38.705319 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:38.705862 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:58:38.706339 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:58:38.706423 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:38.706644 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 20:58:38.706832 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11296,"thread_start_us":11159,"threads_started":1}
I20260929 20:58:38.707139 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:58:38.687332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55420 (local address 127.0.97.61:39507)
0929 20:58:38.687623 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 20:58:38.687630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:58:38.700173 (+ 12543us) server_negotiation.cc:407] Connection header received
0929 20:58:38.700868 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:58:38.700874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:58:38.700958 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:58:38.701017 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:58:38.701902 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:38.702995 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:38.704853 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:58:38.705358 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:58:38.705493 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:58:38.706336 (+ 843us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:58:38.706446 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:58:38.706527 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 20:58:38.706819 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 20:58:38.706904 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260929 20:58:38.712045 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 23138
I20260929 20:58:38.713269 23200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:58:38.823166 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 23138
W20260929 20:58:38.839718 23303 connection.cc:535] server connection from 127.0.0.1:39798 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 20:58:38.840116 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 23212
I20260929 20:58:38.841216 23274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:58:38.947058 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 23212
I20260929 20:58:38.963447 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu pid 23286
I20260929 20:58:38.964529 23348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:58:39.072204 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPL7Duu/build/asan/bin/kudu with pid 23286
2026-09-29T20:58:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4871 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9049 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (349598 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST